{"id":"2d978f48-5744-458d-b940-19c01cf27ef2","arxiv_id":"2505.11980","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AoP-SAM trains a lightweight prompt predictor on SAM's own image embeddings to emit a point-prompt confidence map, then filters redundant prompts at test time, improving automatic segmentation accuracy and efficiency.","lead":"AoP-SAM is a method that automatically generates point prompts for the Segment Anything Model (SAM), so no human clicks or separate object detectors are needed. A lightweight predictor trained on SA-1B plus a test-time filtering step improves both segmentation accuracy and inference speed on COCO and LVIS benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hyperparameters in Table 1 appear to be selected on the very evaluation benchmarks they are meant to generalize to; the paper reports no validation split, so the 'consistently highest mIoU' claim may be an artifact of test-set tuning.","rationale":"The reader's weakest assumption concerns the transferability of SA-1B point prompts to other domains. That is a reasonable concern, but the COCO and LVIS results already provide external evidence that the Prompt Predictor transfers; the SA-1B proxy worry does not directly challenge the reported numbers. By contrast, the absence of a validation split and the discrepancies in hyperparameter selection strike at the core of the empirical claim: if the hyperparameters were tuned on the test sets, the consistently-highest-mIoU result is guaranteed by construction, not by the method. This is a more fundamental threat to the central claim than any implementation detail or notational ambiguity. I therefore do not adopt the reader's weakest assumption as the single load-bearing concern; instead I identify the evaluation protocol. The concrete test above would settle whether the superiority is real.","tokens_in":11441,"tokens_out":12853,"duration_ms":120685,"concrete_test":"Re-run the main experiments (Table 1) after first selecting all four hyperparameters (Smoothing Factor, Confidence Intensity Threshold, Prompt Spacing Factor, Prompt Elimination Threshold) on a held-out validation split of SA-1B (e.g., 10% of the test set), then freeze them for COCO and LVIS without re-tuning. If the reported mIoU margins over OAS(Box)/OAS(Central) shrink or disappear on any dataset, the 'consistently highest mIoU' claim is not established. Release the exact splits and tuning procedure so the selection can be audited.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 'Implementation Details' fixes Smoothing Factor=2, Confidence Intensity Threshold=0.2, and Prompt Spacing Factor=2. Table 3a-d reports ablation sweeps over exactly these hyperparameters (plus the elimination threshold) but does not state the dataset used for the sweeps. In the same section the paper says AoP-SAM is trained on a subset of SA-1B and tested on a separate test set; no validation split is mentioned. If the values in Table 3 were chosen to maximize mIoU on the SA-1B, COCO, and LVIS test sets, then Table 1 compares a method whose hyperparameters are tuned on the test sets against baselines whose hyperparameters are fixed. This would inflate the reported mIoU margins and undermine the claim that AoP-SAM 'consistently achieves the highest mIoU scores'. Furthermore, Table 3c sweeps Prompt Spacing Factor over 4,5,6,7, while Implementation Details sets it to 2, which is outside the ablated range; this inconsistency suggests the reported configuration may not be a result of the presented ablation, or that a different dataset/metric was used. Together, these issues make the empirical superiority of the central claim unverifiable from the paper as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AoP-SAM, a method to automate point-prompt generation for the Segment Anything Model (SAM). A lightweight Prompt Predictor takes the raw image and SAM's image embedding as inputs and produces a Prompt Confidence Map; during training, SA-1B point prompts are used as supervision. At test time, prompts are sampled from the confidence map and then refined by an Adaptive Sampling and Filtering (ASF) mechanism that uses previously generated masks to estimate a Prompt Elimination Map, removing candidates predicted to yield duplicate masks. The method is evaluated on SA-1B, COCO, and LVIS with three SAM encoders (MobileSAM, ViT-L, ViT-H), comparing against grid-based AMG and YOLOv8-based Object-Aware Sampling (OAS), plus an edge-device experiment on an Nvidia Jetson Orin Nano. The central claim is that AoP-SAM consistently achieves the highest mIoU while lowering latency and memory.","tokens_in":11685,"tokens_out":3527,"duration_ms":35201,"significance":"If the empirical results hold, AoP-SAM is a practically useful contribution: it removes manual prompting and the need for an external detector, reuses SAM's image embeddings, and reports gains across multiple encoders and datasets. The COCO and LVIS results are particularly valuable because they are outside the SA-1B training distribution, providing independent evidence for the generality of the prompt predictor. The paper also includes an edge-device efficiency study, which is relevant for deployment. However, the strength of these claims is currently undermined by reproducibility concerns around hyperparameter selection and by an inconsistency in the definition of the elimination threshold.","major_comments":[{"comment":"The paper does not report any validation split or the dataset used for the hyperparameter sweeps in Table 3a-d. Implementation Details fixes Smoothing Factor=2, Confidence Intensity Threshold=0.2, and Prompt Spacing Factor=2, and Table 3a-c sweeps exactly these parameters, but it is not stated whether the choices were made on a validation split or on the SA-1B/COCO/LVIS test sets. If the values were selected to maximize mIoU on the same test sets used in Table 1, then the comparison is biased in favor of AoP-SAM relative to baselines with fixed hyperparameters, and the 'consistently highest mIoU' claim is not verifiable from the text. Please specify the validation protocol, report the split used for tuning, and confirm that Table 1 reports performance on held-out test data.","section":"Experiments, Implementation Details and Table 3"},{"comment":"The chosen Prompt Spacing Factor in Implementation Details is 2, but the ablation in Table 3c only sweeps the values 4, 5, 6, and 7. The reported configuration therefore falls outside the ablated range, so the reader cannot see how the final value was selected or whether it is supported by the ablation. Please either include the chosen value in the sweep or clarify the relationship between the ablation results and the configuration used in Table 1.","section":"Table 3c and Implementation Details"},{"comment":"Equation (3) defines T_elim as (1/n) Σ IoU_i × C_i, where C_i are per-pixel Prompt Elimination Maps defined in Eq. (1). This mixes scalar IoU scores with spatial maps and yields a map, not a scalar threshold; however, the method text and Table 3d treat T_elim as a scalar elimination threshold with values such as 1.25–1.4. Please define the reduction from map to scalar explicitly (for example, by averaging over spatial locations or by evaluating C_i at the candidate prompt positions) and make Eq. (3) dimensionally consistent with its use.","section":"Method, Eq. (3)"},{"comment":"The text states that 'the best performance is observed when both Adaptive Sampling and Adaptive Filtering are used together,' but Table 2 shows that on SA-1B the configuration with Adaptive Sampling only reaches mIoU 72.8, while the full configuration with Adaptive Filtering reaches 71.3. This directly contradicts the stated conclusion for the SA-1B column. Please correct the claim or explain the discrepancy, and similarly reconcile the COCO and LVIS columns if the same issue occurs there.","section":"Table 2, Component Analysis"}],"minor_comments":[{"comment":"The sentence 'We employed a learning rate of and trained the model for 1000 epochs' is missing the numerical value of the learning rate. Please insert the value.","section":"Training of Prompt Predictor"},{"comment":"Several latency entries appear to have missing separators, e.g., '0.0962.1' in the MobileSAM row; these should be formatted as two separate numbers (latency and peak memory) so the table is legible.","section":"Table 1"},{"comment":"The PeakMem column in Table 3a reports values such as 51.5, 42.2, 32.7, and 24.6, while Table 1 reports peak memory in GB with values around 4–7. Please clarify the units in Table 3a or align them with Table 1.","section":"Table 3a"},{"comment":"References [Zhang et al. 2023a] and [Zhang et al. 2023b] both cite the same Faster SAM paper and the citation for Object-Aware Sampling appears to point to MobileSAMv2; please verify that all citations correspond to the intended works.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The central idea is reasonable and the multi-dataset evaluation is a strength, but the missing validation protocol and the threshold definition issue are serious enough that the current version should not be accepted without addressing them. I would encourage the editor to ask the authors for code or a detailed reproducibility appendix, because the hyperparameter-selection concern cannot be fully resolved from the text alone."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper presents a genuinely useful idea — a lightweight prompt predictor that learns from SAM's own image embedding plus the raw image, with a coarse-to-fine adaptive sampler/filter at test time. The main tables are mostly consistent with the claim that it beats both grid-based AMG and YOLOv8-based OAS on SA-1B, COCO, and LVIS across three SAM variants, and the COCO/LVIS results are genuinely external to the SA-1B training distribution, which gives some independent support. The edge-device numbers are a nice bonus.\n\nThat said, the empirical reporting has real problems, and one is hard to ignore. The implementation details set Smoothing Factor=2, Confidence Intensity Threshold=0.2, and Prompt Spacing Factor=2. But the ablations in Table 3 show the best mIoU in those sweeps at 1, 0.1, and 4, respectively. So the reported configuration is not the winner of its own ablations. This doesn't prove the authors tuned on the test set — if they had, they'd have picked the sweep winners — but it does mean we can't tell whether Table 1 comes from the configuration described, or whether the ablations were run on a different dataset or metric. No validation split is ever mentioned, and the ablation dataset is not stated. That's a reproducibility gap, not a fatal flaw.\n\nOther soft spots are smaller. The learning rate is literally missing from the text (\"we employed a learning rate of and trained for 1000 epochs\"). Eq. 3 mixes per-pixel Prompt Elimination Maps with per-mask IoU scalars, so as written Telim is a map, while the prose treats it as a scalar threshold; that needs cleanup. No code, weights, or error bars are provided, which on a systems paper like this is less than ideal.\n\nOn the merits, the novelty is real: no prior work in the cited literature learns prompt confidence maps directly from SAM embeddings with this kind of test-time filtering. The method is simple, plausible, and well matched to the problem. I'd want to see a revision that releases code, clarifies the hyperparameter story, adds a validation split, and fixes the notation. If those hold up, this is a solid systems contribution worth having in the literature.\n\nFor peer review: yes, send it out. It deserves a serious referee, but only with a request for major revision on the reporting. I'd cite it with a caveat in my own work.","headline":"AoP-SAM is a genuinely useful idea — a lightweight prompt predictor that leverages SAM's own embeddings — and the main experimental claims mostly hold up; but the paper's internal hyperparameter reporting is inconsistent enough that nobody can reproduce the numbers as written.","tokens_in":12258,"tokens_out":4503,"would_cite":true,"duration_ms":43065,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AoP-SAM aims to make SAM fully automatic by learning to place point prompts from SAM's own image embeddings and filtering redundant candidates at test time, claiming the highest mIoU on three datasets with lower latency and memory than…","keywords":["Segment Anything Model","automatic prompt generation","prompt confidence map","adaptive sampling and filtering","test-time adaptation","image segmentation","zero-shot segmentation","edge inference"],"falsifier":"A decisive check is to fix the prompt budget and compare AoP-SAM against random point prompts of the same count and against oracle points drawn from inside ground-truth masks on COCO and LVIS: if the learned confidence map does not clearly beat random placement, the predictor is not doing the work; if it approaches the oracle, prompt placement is the source of the gain.","tokens_in":11210,"feed_emoji":"🎯","tokens_out":7412,"duration_ms":66658,"temperature":0.7,"pith_summary":"The paper seeks to establish that the Segment Anything Model can be made fully automatic—no human clicks and no external object detector—by learning where to put point prompts and then pruning redundant ones during inference. Its AoP-SAM pipeline consists of a lightweight Prompt Predictor that consumes both the input image and SAM's own ViT image embedding to emit a Prompt Confidence Map, followed by a test-time Adaptive Sampling and Filtering (ASF) mechanism that converts the map into sparse point prompts and removes candidates that would reproduce an already-generated mask. The paper reports that AoP-SAM achieves the highest mIoU on SA-1B, COCO, and LVIS across MobileSAM, ViT-L, and ViT-H encoders, while using lower latency and peak memory than grid-search AMG and YOLOv8-based Object-Aware Sampling. If correct, the work turns SAM into a turnkey segmenter for annotation and edge deployment because it reuses SAM's expensive image embedding and keeps the prompt model small.","feed_headline":"Learned prompt placement beats grid and detector prompts for SAM","feed_subtitle":"It reports higher mIoU with lower latency on three datasets, including on an edge GPU.","key_machinery":"The load-bearing machinery is the Prompt Confidence Map produced by the Prompt Predictor, a lightweight CNN with two encoders (one for the image, one for the reshaped ViT embedding) and a sigmoid decoder. The map is the single object that connects prompt prediction to SAM: local maxima of the smoothed map give the coarse prompt pool, and the map's feature space is also reused by the Adaptive Sampling and Filtering module, which builds a Prompt Elimination Map from the cosine similarity between pixel-wise L2-normalized image features and average-pooled mask features. This elimination map, combined with IoU-weighted thresholds, is what lets the method avoid redundant mask refinements and keep only essential prompts.","core_discovery":"On the paper's own terms, the central discovery is that the information SAM already computes for every image—the image embedding from its ViT encoder—contains enough signal to predict where a point prompt will produce a useful mask, so prompt automation can live inside SAM instead of in a separate detector. The Prompt Predictor is a small CNN that takes the raw image and the reshaped ViT embedding, fuses them, and outputs a sigmoid Prompt Confidence Map; local maxima of the smoothed map become prompt candidates. Training on SA-1B point prompts with an L2 (MSE) loss teaches the predictor to imitate SAM's own notion of essential prompt locations. At test time, ASF samples candidates coarsely, generates masks, and then computes a Prompt Elimination Map by cosine similarity between L2-normalized image features and pooled mask features; prompts whose elimination scores exceed an IoU-weighted threshold are discarded as redundant. The paper's headline evidence is that this two-stage mechanism yields the highest mIoU on all three datasets with all three encoders, and that on an edge GPU it cuts prompt-production latency and peak memory relative to the detector-based baseline.","pith_inferences":["One testable extension is to replace the SA-1B prompt target with ground-truth mask centers on COCO/LVIS and see whether the learned confidence map shifts; if it does not, the SA-1B-trained predictor is already domain-robust, and if it does, a lightweight domain-adaptive fine-tune could close the gap.","The same Prompt Elimination Map idea could be used as a stopping criterion: when the elimination map covers most remaining candidates, iteration can halt, saving further latency beyond prompt filtering.","A deeper implication is that any prompt-based segmenter that exposes image and mask features could adopt the coarse-to-fine elimination scheme, not only SAM-family models.","The authors do not report what happens when the prompt budget is fixed to match the best baseline; measuring mIoU at equal prompt counts would isolate the contribution of prompt placement from the contribution of filtering."],"forward_implications":["SAM can be deployed for fully automatic segmentation without any manual clicking or separate detection model, removing the need for human prompt engineering in annotation pipelines.","Because the Prompt Predictor reuses SAM's image embedding, the marginal cost of prompt generation is small, so the method is suited to latency- and memory-constrained settings such as edge GPUs.","Using fewer, better-targeted prompts reduces redundant mask generations, which directly lowers both inference latency and peak memory during mask decoding.","The frozen-SAM design means zero-shot generalization is preserved; the prompt predictor inherits SA-1B's coverage without fine-tuning SAM itself.","The test-time filtering adapts per image to the masks already produced, so objects of very different sizes can share the same prompt budget without a dense grid."],"supporting_citations":[{"why":"Supplies SAM itself, the frozen image/prompt encoder and mask decoder, the SA-1B dataset used for training the Prompt Predictor, and the grid-based AMG baselines.","marker":"Kirillov et al. 2023"},{"why":"Defines the Object-Aware Sampling baseline that uses YOLOv8 bounding boxes to generate box and central-point prompts, which is the main efficiency comparison.","marker":"Zhang et al. 2023c"},{"why":"Provides MobileSAM, the lightweight encoder used for the edge-device efficiency experiment.","marker":"Zhang et al. 2023b"},{"why":"Supplies the COCO dataset used as one of the three evaluation benchmarks.","marker":"Lin et al. 2014"},{"why":"Supplies the LVIS long-tail dataset used as the third evaluation benchmark.","marker":"Gupta, Dollar, and Girshick 2019"},{"why":"Supplies the greedy IoU matching protocol used to compute mIoU for prompt-based mask generation.","marker":"Zhang et al. 2023d"},{"why":"Supplies the Prompt Elimination Map idea that the ASF module adapts for filtering redundant prompts.","marker":"Zhang et al. 2023e"}],"fun_headline_variants":["Learned prompt placement inside SAM beats grid and detector prompts","SAM auto-generates prompts from its own embeddings, cutting latency","Automated coarse-to-fine prompt sampling improves SAM efficiency","No manual prompts: AoP-SAM learns optimal click locations for SAM","SAM's ViT embedding guides prompt automation: higher mIoU, less compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the SA-1B point prompts used as ground truth encode where effective prompts should go; if that distribution does not transfer to COCO, LVIS, or real scenes, the Prompt Confidence Map will point SAM at the wrong places.","fun_headline_variants_meta":{"raw":{"variants":["Learned prompt placement inside SAM beats grid and detector prompts","SAM auto-generates prompts from its own embeddings, cutting latency","Automated coarse-to-fine prompt sampling improves SAM efficiency","No manual prompts: AoP-SAM learns optimal click locations for SAM","SAM's ViT embedding guides prompt automation: higher mIoU, less compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001005,"raw_usage":{"total_tokens":4261,"prompt_tokens":970,"completion_tokens":3291,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":3201}},"tokens_in":586,"tokens_out":3291,"duration_ms":23859,"temperature":1.0,"reasoning_tokens":3201,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:43:38.922186+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check is to fix the prompt budget and compare AoP-SAM against random point prompts of the same count and against oracle points drawn from inside ground-truth masks on COCO and LVIS: if the learned confidence map does not clearly beat random placement, the predictor is not doing the work; if it approaches the oracle, prompt placement is the source of the gain.","supporting_citations":[],"review_version":1}