{"id":"770b4d17-e703-412d-8d81-3ebd7cc7c3c6","arxiv_id":"2412.10032","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Object-Focused Data Selection clusters foundation-model object features per class and selects representative images, outperforming random selection and eight baselines for detection and segmentation under annotation budgets.","lead":"Object-Focused Data Selection (OFDS) picks images for labeling by clustering object-level features from foundation models, per target class, instead of using whole-image similarity. It beats random selection and eight baselines for object detection and segmentation in class-imbalanced settings, and can warm-start active learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core mechanism presumes the object proposer has high recall for all target classes, but recall is never measured; low recall on rare or novel classes would systematically exclude those objects from selection.","rationale":"The reader's weakest_assumption identifies the same concern: false negatives in the object proposer undermine the semantic coverage claim. The paper's own Limitations section flags this dependence, but the experiments do not quantify recall, and the method's success on PASCAL VOC CI and Cityscapes may be explained by the fact that Grounding DINO detects those common classes well. A secondary issue is that the abstract's 'consistently outperforms' is contradicted by Table 7 at several balanced budgets, but those differences are small and could be within noise, so I do not elevate that here. The false-negative issue is more fundamental: it determines whether the method can generalize beyond datasets where the proposer is strong. The proposed test—measuring per-class recall and simulating degraded recall—would settle whether the concern actually lands. Since the reader already conditioned acceptance on additional evidence and this concern is acknowledged, the verdict need not change.","tokens_in":29928,"tokens_out":11853,"duration_ms":134544,"concrete_test":"On a random 200-image subsample of each target pool (PASCAL VOC CI, Cityscapes, LoveDA), obtain dense human labels and compute Grounding DINO's per-class recall at the exact threshold used in the paper (5% FPR calibration; App. I). Report recall separately for rare classes and for small objects (<0.05% area). Then run a controlled simulation: run OFDS on the full pool, but delete a fraction of detections from the rare classes to match the measured recall; train the downstream models and compare to full-recall OFDS. If the performance gap is comparable to the reported OFDS-vs-random margin, the false-negative assumption is load-bearing; if not, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"OFDS's central claim is that clustering object-level features and selecting one object per cluster yields semantic coverage of all target classes. This requires that every important object is present in the proposal set (Algorithm 1, Step 1, Sec. 3.2). The confidence threshold is calibrated to a 5% false-positive rate, deliberately trading recall for precision, and the paper's Limitations states that OFDS 'inherits [the object proposer's] biases and limitations.' Yet no experiment reports the proposer's recall on the target pools. On PASCAL VOC CI and Cityscapes the rare classes are common COCO categories for which Grounding DINO recall is likely high; LoveDA, where the domain is specialized and the threshold is borrowed from Cityscapes (App. I.1), shows smaller margins. Additionally, App. J filters out proposals smaller than 0.05% of image area, removing small objects that are often rare and hard. If a target class has low recall, clusters for that class are empty or incomplete, and the selected subset can miss that class entirely, while random selection would still include some of its images. The central 'coverage' mechanism therefore collapses exactly in the settings the paper targets (rare classes, specialized domains), unless recall is verified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Object-Focused Data Selection (OFDS), a single-pass, training-free method for selecting images to label for object detection and segmentation under a budget measured in annotation units. OFDS uses Grounding DINO to propose objects, SAM 2 to extract object-level features, per-class adaptive k-means clustering, and selects one object per cluster so that the chosen images semantically cover each target class. The method is evaluated on PASCAL VOC (including a deliberately class-imbalanced variant), Cityscapes, and LoveDA, against eight single-pass baselines, across multiple downstream architectures, and as a warm start for active learning. The authors report that OFDS consistently outperforms random selection and baselines, especially under class imbalance, and that combining OFDS-selected human labels with foundation-model autolabels gives the best results at constrained budgets.","tokens_in":30133,"tokens_out":5272,"duration_ms":57815,"significance":"If the empirical claims hold, OFDS addresses a practical and under-studied problem: selecting annotation units for dense prediction tasks before any downstream model is trained. The paper has several genuine strengths: it moves data selection from image-level to object-level features, explicitly models annotation-unit budgets, evaluates a broad set of baselines across multiple tasks and datasets, includes ablations on the clustering step, object proposer strength, downstream architectures, and repeated random/FreeSel runs for one dataset, and provides class-balance analysis of the selected subsets. The object-level coverage idea is intuitive and the imbalanced-setting results are consistently favorable. However, the central claim of universal superiority over random selection is not fully supported by the paper's own balanced-setting tables, and the mechanism's reliance on object-proposer recall is not tested, which is a load-bearing risk for the rare-class and specialized-domain scenarios the paper emphasizes.","major_comments":[{"comment":"The abstract states that OFDS \"consistently outperforms random selection and all baselines,\" but the paper's own balanced-setting results do not support this. In Table 7 (semantic segmentation on the original balanced PASCAL VOC split), OFDS has mIoU 48.52 versus random 49.60 at the 10% budget and 61.89 versus 61.93 at the 40% budget; at several other budgets the differences are small and not consistently in OFDS's favor. The main-text discussion in Sec. 4.2 correctly acknowledges that OFDS performs \"on par\" with random on full PASCAL VOC. Please revise the abstract and conclusion to state the qualified version of the claim.","section":"Abstract; Sec. 4.2; Table 7 in App. Q"},{"comment":"The coverage mechanism presumes that the object proposer has high recall for every target class, but recall is never measured. The confidence threshold is calibrated to a 5% false-positive rate (App. I.2), which deliberately trades recall for precision, and the Limitations section concedes that OFDS \"inherits [the object proposer's] biases and limitations.\" For any class that the proposer systematically misses, the per-class feature set in Step 5 of Algorithm 1 is empty or incomplete, so the selected subset can omit that class entirely, while random selection would still include some of its images. This is precisely the risk for the rare-class and specialized-domain settings the paper targets. Please report per-class recall of the object proposer on the target validation sets and, if necessary, adjust the selection procedure or threshold so that all target classes are verifiably present in the selected subset.","section":"Algorithm 1, Step 1; Sec. 3.2; App. I.2; Limitations"},{"comment":"The budget update in Eq. (2) depends on N_O, the estimated average number of target objects per image, but the main text does not state how N_O is obtained or validated. The reported actual subset sizes are close to the target sizes, which suggests N_O is set from data statistics, but no estimation procedure or sensitivity analysis is provided. Since an inaccurate N_O directly changes the number of images selected and the per-class budget distribution, please document how N_O is computed for each dataset and include an ablation or robustness check.","section":"Eq. (2); Algorithm 1 input"},{"comment":"The implementation paragraph says that \"we only consider object proposals with bounding boxes smaller than 0.05% of the overall image area to filter out noisy proposals.\" Interpreted literally, this removes small proposals, which are often exactly the rare and difficult objects the method aims to cover. If the intended condition is that proposals larger than this threshold are removed, please correct the text; otherwise, justify why filtering out small proposals is consistent with the claimed semantic coverage of all target classes.","section":"App. J (Implementation Details)"}],"minor_comments":[{"comment":"The sentence \"they only work in the image feature space\" contains a typo and should read \"they only work in the image feature space.\"","section":"Sec. 4.1, Baselines paragraph"},{"comment":"The term \"annotation units\" is used for both bounding boxes and masks, but for semantic segmentation multiple masks on the same image can count separately, as noted in App. H.1. Please define this counting convention consistently when the annotation budget is introduced in Sec. 3.","section":"Sec. 3.2; App. H.1"},{"comment":"The adaptive cluster search multiplies the number of clusters by 1.05 until enough clusters are found, but the choice of the factor is not motivated or ablated. A brief discussion of its influence would help.","section":"App. J, adaptive cluster search"},{"comment":"Repeated selection runs are reported only for PASCAL VOC with class imbalance. Since random selection and FreeSel are stochastic, it would be helpful to state that single runs are used elsewhere or to provide repeated-run intervals for at least one additional dataset.","section":"App. F"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and the experiments are extensive, but the abstract overclaims relative to the balanced-setting results, and the unmeasured object-proposer recall is a genuine correctness risk for the method's central coverage mechanism. I believe these issues can be fixed within a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: OFDS is a real contribution to single-pass data selection for detection and segmentation. It is the first method I know that operates at object level with annotation-unit budgets, and the imbalanced-class experiments are consistently in its favor. But the abstract overclaims: on balanced PASCAL VOC segmentation, random selection ties or beats OFDS at several budget points, so the \"consistently outperforms\" sentence should be softened. And the paper never measures recall of the object proposer, which is the load-bearing assumption behind the coverage story.\n\nWhat's genuinely new: using SAM2 object features, per-class adaptive k-means, and a budget expressed in annotation units rather than images. The ablation replacing clustering with per-class random proposal selection shows a small but consistent benefit, and the combination with autolabeling is a sensible practical recipe. The experiments cover four datasets, two downstream tasks (plus instance segmentation), two downstream architectures, and an active-learning warm-start. That is a solid empirical base.\n\nSoft spots, in proportion: first, the balanced-setting overstatement is real, and it is in the abstract, not just a loose phrase. Table 7 shows random ahead of OFDS at several fractions. Second, the recall concern: the 5% FPR threshold on a reference set controls precision, not recall, and the paper never reports recall on target pools. For rare classes the proposer misses, clustering has no objects to cover, and random selection would still include some of those images by chance. LoveDA's smaller margin is consistent with this worry. That does not kill the method—the imbalanced results are strong—but recall should be measured before claiming semantic coverage. Third, N_O in Eq. 2 is never specified, and there are no error bars on OFDS itself (only random and FreeSel are repeated). These are fixable, not fatal.\n\nWho should read this: anyone working on data-centric AI, active-learning cold starts, or annotation-efficient dense prediction. It deserves a serious referee; I would want to see recall/coverage numbers, repeated runs, and a corrected abstract before accepting.\n\nRecommendation: send it to review, with a request for revision on those points.","headline":"OFDS is a genuine object-level data selection method with strong imbalanced-class results, but the abstract overstates consistency on balanced settings and the paper never measures the object proposer's recall, which the coverage story depends on.","tokens_in":30720,"tokens_out":2657,"would_cite":true,"duration_ms":26133,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By clustering foundation-model object features per class and choosing one object per cluster, Object-Focused Data Selection beats random selection and eight baselines for detection and segmentation labeling budgets.","keywords":["data selection","object detection","semantic segmentation","foundation models","class imbalance","annotation budget","active learning","k-means clustering"],"falsifier":"Evaluate OFDS on a dataset where a target class is present but reliably missed by the object proposer (e.g., a specialized remote-sensing class with low recall at the 5% false-positive threshold). If training on OFDS-selected subsets does not beat random selection for that class, it confirms the proposed selection inherits proposer recall failures. A sharper test: replace Grounding DINO proposals with oracle human-annotated boxes as the proposer input; if downstream performance does not improve over the standard proposer, then the bottleneck is elsewhere (e.g., clustering or features), not proposal recall.","tokens_in":29696,"feed_emoji":"🎯","tokens_out":5383,"duration_ms":52038,"temperature":0.7,"pith_summary":"The paper asks which unlabeled images to send to human annotators when building a training set for object detection or segmentation under a fixed annotation budget. It claims that a single-pass selection method operating on object-level features, rather than whole-image features, can reliably beat random selection and eight existing baselines. The key move is to cluster foundation-model object features per target class and pick one object from each cluster, so that the selected images cover the semantic variety of every class. This matters because annotation budgets are costly and real-world data is class-imbalanced; the paper reports consistent gains, with the largest gains on rare classes and on several datasets. It also shows that combining these human labels with foundation-model autolabels gives the best results under tight budgets.","feed_headline":"Clustering objects, not images, picks better data to label","feed_subtitle":"Object-level selection beats random sampling for detection and segmentation, especially on rare classes.","key_machinery":"The load-bearing mechanism is class-wise adaptive k-means clustering of foundation-model object features. Each cluster represents a semantically coherent group of objects within a target class; selecting the object nearest to each cluster center yields a density-based covering of that class's feature space, while processing classes by ascending proposal count protects rare classes. The adaptive k (Eq. 2) converts the remaining annotation budget into a number of clusters, and the search for clusters without already-annotated images prevents near-duplicates.","core_discovery":"OFDS selects images to label by first running an open-world detector (Grounding DINO) to propose objects, extracting 256-dimensional object features from SAM 2, and assigning each proposal to a target class by its predicted label. For each class it clusters those features with k-means, choosing k adaptively from the remaining annotation budget and the number of already selected images, and then selects one object nearest to each cluster centroid. The images containing those selected objects are exhaustively labeled by humans. Because classes are processed in ascending order of proposal count, rare classes get covered first. The paper reports that this object-level semantic covering consistently outperforms random selection and all baselines across object detection, semantic segmentation, and instance segmentation on PASCAL VOC (balanced and artificially imbalanced), Cityscapes, and LoveDA, and that the margin is largest for rare classes and for class-imbalanced settings.","pith_inferences":["Because OFDS relies entirely on the object proposer's recall, its gains should scale with foundation-model improvements; a better proposer should translate directly into better selection, and the paper's own ablation with a larger backbone supports this extrapolation.","The method suggests a testable recipe for other dense prediction tasks: any task whose annotation unit is an object-like region (e.g., panoptic parts, medical lesions) could use the same cluster-and-cover selection, provided a proposer and feature extractor exist for that domain.","The near-duplicate-removal property of cluster-center selection may be valuable beyond annotation budgeting, e.g., in cleaning web-scale image collections before self-supervised pre-training, though the paper only demonstrates the duplication-robustness in a synthetic duplicate setting.","The authors do not claim OFDS replaces active learning; their direct comparison shows it is on par with the tested active learning frameworks while being model-agnostic, which suggests data selection and active learning are complementary rather than competing stages."],"forward_implications":["Under a fixed annotation budget measured in annotation units, training a compact detector or segmenter on an OFDS-selected subset yields higher mAP or mIoU than training on subsets selected by random drawing or by any of the eight image-level baselines tested.","The advantage over baselines is largest for the rarest classes, meaning OFDS mitigates class imbalance at the data-selection stage rather than through training-time loss adjustments.","Pre-training a downstream model on full-dataset autolabels from foundation models and then fine-tuning on an OFDS-selected human-labeled subset outperforms either autolabels alone or human labels alone for the same annotation cost.","Using OFDS to choose the initial labeled set for active learning improves the entire active learning curve compared with a random cold start or HaCON.","OFDS is deterministic and requires no labeled data and no downstream model training during selection; selection is a single pass with cost comparable to one forward inference pass."],"supporting_citations":[{"why":"Grounding DINO is the open-world object proposer that generates the initial bounding-box proposals and class predictions for OFDS.","marker":"[31]"},{"why":"SAM 2 supplies the 256-dimensional object features (memory-bank pointers) that OFDS clusters per class.","marker":"[37]"},{"why":"FreeSel is the closest image-level single-pass baseline; OFDS is compared against it on all datasets and in the autolabel fine-tuning experiments.","marker":"[55]"},{"why":"PASCAL VOC provides the main detection and segmentation benchmarks, including the artificially class-imbalanced variant used for the headline results.","marker":"[15]"},{"why":"Cityscapes provides the naturally class-imbalanced urban-scene benchmark for detection, segmentation, and instance segmentation.","marker":"[9]"},{"why":"LoveDA tests OFDS in a specialized remote-sensing domain where the foundation model proposer is expected to be weaker.","marker":"[50]"},{"why":"HaCON is the cold-start baseline for active learning warm starting; OFDS is compared against it for initial labeled-set selection.","marker":"[7]"},{"why":"Sorscher et al. supply the class-balance score used to measure the balance of selected subsets and the prototype-selection method that OFDS outperforms.","marker":"[45]"}],"fun_headline_variants":["Object clusters beat image features for labeling data","Rare classes first: smarter data selection for labeling","OFDS uses object-level features to select better data","Target-object focus improves data selection for detection","Clustering object features picks data that boosts segmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole selection rests on the object proposer's proposals and the SAM2 features accurately representing the true objects of every target class; if the proposer misses a class or an object appearance, that class or appearance is never clustered and never selected, because OFDS only controls the false-positive rate, not the false-negative rate.","fun_headline_variants_meta":{"raw":{"variants":["Object clusters beat image features for labeling data","Rare classes first: smarter data selection for labeling","OFDS uses object-level features to select better data","Target-object focus improves data selection for detection","Clustering object features picks data that boosts segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1268,"prompt_tokens":870,"completion_tokens":398,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":327}},"tokens_in":486,"tokens_out":398,"duration_ms":4989,"temperature":1.0,"reasoning_tokens":327,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:25:16.077246+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate OFDS on a dataset where a target class is present but reliably missed by the object proposer (e.g., a specialized remote-sensing class with low recall at the 5% false-positive threshold). If training on OFDS-selected subsets does not beat random selection for that class, it confirms the proposed selection inherits proposer recall failures. A sharper test: replace Grounding DINO proposals with oracle human-annotated boxes as the proposer input; if downstream performance does not improve over the standard proposer, then the bottleneck is elsewhere (e.g., clustering or features), not proposal recall.","supporting_citations":[{"cited_title":"Grounding dino: Marrying dino with grounded pre-training for open-set object detection","cited_arxiv_id":null,"evidence_quote":"Grounding DINO is the open-world object proposer that generates the initial bounding-box proposals and class predictions for OFDS."},{"cited_title":"Towards free data selection with general-purpose models","cited_arxiv_id":null,"evidence_quote":"FreeSel is the closest image-level single-pass baseline; OFDS is compared against it on all datasets and in the autolabel fine-tuning experiments."},{"cited_title":"Williams, John Winn, and Andrew Zisserman","cited_arxiv_id":null,"evidence_quote":"PASCAL VOC provides the main detection and segmentation benchmarks, including the artificially class-imbalanced variant used for the headline results."},{"cited_title":"Loveda: A remote sensing land-cover dataset for domain adaptive semantic segmentation","cited_arxiv_id":null,"evidence_quote":"LoveDA tests OFDS in a specialized remote-sensing domain where the foundation model proposer is expected to be weaker."},{"cited_title":"Making your first choice: To address cold start problem in medical active learning","cited_arxiv_id":null,"evidence_quote":"HaCON is the cold-start baseline for active learning warm starting; OFDS is compared against it for initial labeled-set selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Sorscher et al. supply the class-balance score used to measure the balance of selected subsets and the prototype-selection method that OFDS outperforms."}],"review_version":1}