Pith. sign in

REVIEW 4 major objections 4 minor 25 references

AoP-SAM: Automation of Prompts for Efficient Segmentation

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read AoP-SAM aims to make SAM fully automatic by learning to place point prompts from SAM's own image embeddings and filtering redundant candidates at test time, claiming the highest mIoU on three datasets with lower latency and memory than…

desk verdict AoP-SAM is a genuinely useful idea — a lightweight prompt predictor that leverages SAM's own embeddings — and the main experimental claims mostly hold up; but the paper's internal hyperparameter reporting is inconsistent enough that nobody can reproduce the numbers as written. read the letter →

arxiv 2505.11980 v1 pith:HSBIS4FH submitted 2025-05-17 cs.CV cs.AI

classification cs.CVcs.AI
keywords SegmentAnythingModelautomaticpromptgenerationconfidencemapadaptivesamplingandfilteringtest-timeadaptationimagesegmentationzero-shotedgeinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper seeks to establish that the Segment Anything Model can be made fully automatic—no human clicks and no external object detector—by learning where to put point prompts and then pruning redundant ones during inference. Its AoP-SAM pipeline consists of a lightweight Prompt Predictor that consumes both the input image and SAM's own ViT image embedding to emit a Prompt Confidence Map, followed by a test-time Adaptive Sampling and Filtering (ASF) mechanism that converts the map into sparse point prompts and removes candidates that would reproduce an already-generated mask. The paper reports that AoP-SAM achieves the highest mIoU on SA-1B, COCO, and LVIS across MobileSAM, ViT-L, and ViT-H encoders, while using lower latency and peak memory than grid-search AMG and YOLOv8-based Object-Aware Sampling. If correct, the work turns SAM into a turnkey segmenter for annotation and edge deployment because it reuses SAM's expensive image embedding and keeps the prompt model small.

What carries the argument

The load-bearing machinery is the Prompt Confidence Map produced by the Prompt Predictor, a lightweight CNN with two encoders (one for the image, one for the reshaped ViT embedding) and a sigmoid decoder. The map is the single object that connects prompt prediction to SAM: local maxima of the smoothed map give the coarse prompt pool, and the map's feature space is also reused by the Adaptive Sampling and Filtering module, which builds a Prompt Elimination Map from the cosine similarity between pixel-wise L2-normalized image features and average-pooled mask features. This elimination map, combined with IoU-weighted thresholds, is what lets the method avoid redundant mask refinements and keep only essential prompts.

What would settle it

A decisive check is to fix the prompt budget and compare AoP-SAM against random point prompts of the same count and against oracle points drawn from inside ground-truth masks on COCO and LVIS: if the learned confidence map does not clearly beat random placement, the predictor is not doing the work; if it approaches the oracle, prompt placement is the source of the gain.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the information SAM already computes for every image—the image embedding from its ViT encoder—contains enough signal to predict where a point prompt will produce a useful mask, so prompt automation can live inside SAM instead of in a separate detector. The Prompt Predictor is a small CNN that takes the raw image and the reshaped ViT embedding, fuses them, and outputs a sigmoid Prompt Confidence Map; local maxima of the smoothed map become prompt candidates. Training on SA-1B point prompts with an L2 (MSE) loss teaches the predictor to imitate SAM's own notion of essential prompt locations. At test time, ASF samples candidates coarsely, generates masks, and then computes a Prompt Elimination Map by cosine similarity between L2-normalized image features and pooled mask features; prompts whose elimination scores exceed an IoU-weighted threshold are discarded as redundant. The paper's headline evidence is that this two-stage mechanism yields the highest mIoU on all three datasets with all three encoders, and that on an edge GPU it cuts prompt-production latency and peak memory relative to the detector-based baseline.

Load-bearing premise

The load-bearing premise is that the SA-1B point prompts used as ground truth encode where effective prompts should go; if that distribution does not transfer to COCO, LVIS, or real scenes, the Prompt Confidence Map will point SAM at the wrong places.

Editorial extensions

If this is right

  • SAM can be deployed for fully automatic segmentation without any manual clicking or separate detection model, removing the need for human prompt engineering in annotation pipelines.
  • Because the Prompt Predictor reuses SAM's image embedding, the marginal cost of prompt generation is small, so the method is suited to latency- and memory-constrained settings such as edge GPUs.
  • Using fewer, better-targeted prompts reduces redundant mask generations, which directly lowers both inference latency and peak memory during mask decoding.
  • The frozen-SAM design means zero-shot generalization is preserved; the prompt predictor inherits SA-1B's coverage without fine-tuning SAM itself.
  • The test-time filtering adapts per image to the masks already produced, so objects of very different sizes can share the same prompt budget without a dense grid.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is to replace the SA-1B prompt target with ground-truth mask centers on COCO/LVIS and see whether the learned confidence map shifts; if it does not, the SA-1B-trained predictor is already domain-robust, and if it does, a lightweight domain-adaptive fine-tune could close the gap.
  • The same Prompt Elimination Map idea could be used as a stopping criterion: when the elimination map covers most remaining candidates, iteration can halt, saving further latency beyond prompt filtering.
  • A deeper implication is that any prompt-based segmenter that exposes image and mask features could adopt the coarse-to-fine elimination scheme, not only SAM-family models.
  • The authors do not report what happens when the prompt budget is fixed to match the best baseline; measuring mIoU at equal prompt counts would isolate the contribution of prompt placement from the contribution of filtering.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes AoP-SAM, a method to automate point-prompt generation for the Segment Anything Model (SAM). A lightweight Prompt Predictor takes the raw image and SAM's image embedding as inputs and produces a Prompt Confidence Map; during training, SA-1B point prompts are used as supervision. At test time, prompts are sampled from the confidence map and then refined by an Adaptive Sampling and Filtering (ASF) mechanism that uses previously generated masks to estimate a Prompt Elimination Map, removing candidates predicted to yield duplicate masks. The method is evaluated on SA-1B, COCO, and LVIS with three SAM encoders (MobileSAM, ViT-L, ViT-H), comparing against grid-based AMG and YOLOv8-based Object-Aware Sampling (OAS), plus an edge-device experiment on an Nvidia Jetson Orin Nano. The central claim is that AoP-SAM consistently achieves the highest mIoU while lowering latency and memory.

Significance. If the empirical results hold, AoP-SAM is a practically useful contribution: it removes manual prompting and the need for an external detector, reuses SAM's image embeddings, and reports gains across multiple encoders and datasets. The COCO and LVIS results are particularly valuable because they are outside the SA-1B training distribution, providing independent evidence for the generality of the prompt predictor. The paper also includes an edge-device efficiency study, which is relevant for deployment. However, the strength of these claims is currently undermined by reproducibility concerns around hyperparameter selection and by an inconsistency in the definition of the elimination threshold.

major comments (4)
  1. [Experiments, Implementation Details and Table 3] The paper does not report any validation split or the dataset used for the hyperparameter sweeps in Table 3a-d. Implementation Details fixes Smoothing Factor=2, Confidence Intensity Threshold=0.2, and Prompt Spacing Factor=2, and Table 3a-c sweeps exactly these parameters, but it is not stated whether the choices were made on a validation split or on the SA-1B/COCO/LVIS test sets. If the values were selected to maximize mIoU on the same test sets used in Table 1, then the comparison is biased in favor of AoP-SAM relative to baselines with fixed hyperparameters, and the 'consistently highest mIoU' claim is not verifiable from the text. Please specify the validation protocol, report the split used for tuning, and confirm that Table 1 reports performance on held-out test data.
  2. [Table 3c and Implementation Details] The chosen Prompt Spacing Factor in Implementation Details is 2, but the ablation in Table 3c only sweeps the values 4, 5, 6, and 7. The reported configuration therefore falls outside the ablated range, so the reader cannot see how the final value was selected or whether it is supported by the ablation. Please either include the chosen value in the sweep or clarify the relationship between the ablation results and the configuration used in Table 1.
  3. [Method, Eq. (3)] Equation (3) defines T_elim as (1/n) Σ IoU_i × C_i, where C_i are per-pixel Prompt Elimination Maps defined in Eq. (1). This mixes scalar IoU scores with spatial maps and yields a map, not a scalar threshold; however, the method text and Table 3d treat T_elim as a scalar elimination threshold with values such as 1.25–1.4. Please define the reduction from map to scalar explicitly (for example, by averaging over spatial locations or by evaluating C_i at the candidate prompt positions) and make Eq. (3) dimensionally consistent with its use.
  4. [Table 2, Component Analysis] The text states that 'the best performance is observed when both Adaptive Sampling and Adaptive Filtering are used together,' but Table 2 shows that on SA-1B the configuration with Adaptive Sampling only reaches mIoU 72.8, while the full configuration with Adaptive Filtering reaches 71.3. This directly contradicts the stated conclusion for the SA-1B column. Please correct the claim or explain the discrepancy, and similarly reconcile the COCO and LVIS columns if the same issue occurs there.
minor comments (4)
  1. [Training of Prompt Predictor] The sentence 'We employed a learning rate of and trained the model for 1000 epochs' is missing the numerical value of the learning rate. Please insert the value.
  2. [Table 1] Several latency entries appear to have missing separators, e.g., '0.0962.1' in the MobileSAM row; these should be formatted as two separate numbers (latency and peak memory) so the table is legible.
  3. [Table 3a] The PeakMem column in Table 3a reports values such as 51.5, 42.2, 32.7, and 24.6, while Table 1 reports peak memory in GB with values around 4–7. Please clarify the units in Table 3a or align them with Table 1.
  4. [References] References [Zhang et al. 2023a] and [Zhang et al. 2023b] both cite the same Faster SAM paper and the citation for Object-Aware Sampling appears to point to MobileSAMv2; please verify that all citations correspond to the intended works.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AoP-SAM is a learned prompt predictor evaluated on external COCO/LVIS benchmarks; no derivation step reduces to its own inputs or to a self-citation chain.

full rationale

The paper's derivation chain is a supervised learning pipeline: the Prompt Predictor is trained with SA-1B point prompts as ground truth, and its outputs are then consumed by SAM to generate masks, with test-time Adaptive Sampling and Filtering as a heuristic post-process. The reported equations (Eq. 1-3) compute cosine-similarity elimination scores from generated masks and image features; none of them reintroduces the training target as the predicted quantity. The final mIoU numbers come from greedy matching of SAM masks to ground-truth masks, so the predicted confidence map is not itself the evaluation metric. The SA-1B test split is in-domain but is stated to be separate from the training subset, and the central claim of consistent mIoU improvement is also demonstrated on COCO and LVIS, which are external to the SA-1B training distribution. Citations to Zhang et al. for the Prompt Elimination Map and to Kirillov et al. for SAM provide background and prior components, but the paper does not invoke a self-citation uniqueness theorem or adopt a load-bearing ansatz from the authors' own prior work. The hyperparameter sweeps in Table 3 lack a stated validation split, which is an evaluation-reporting weakness rather than a circularity: there is no quoted step where a fitted parameter is renamed as a prediction or where an equation is equivalent to its input by construction. Accordingly, no specific circular step can be identified under the hard-evidence rule.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method rests on the assumption that SA-1B point prompts are a good supervision signal, that SAM's embedding carries enough information for prompt localization, and that feature similarity to a reference mask implies prompt redundancy. No formal proof is offered for the ASF elimination criterion. Four hand-tuned hyperparameters are selected via ablation, and no new physical or conceptual external entities are introduced beyond the trained predictor's output maps.

free parameters (4)
  • Sampling Smoothing Factor = 2 (selected via ablation in Table 3a)
    Controls Gaussian smoothing of the prompt confidence map before local-maxima extraction; values 1 to 4 are swept and the default is 2.
  • Confidence Intensity Threshold = 0.2 (selected via ablation in Table 3b)
    Absolute threshold for accepting local maxima as prompt candidates; values 0.1 to 0.4 are swept.
  • Prompt Spacing Factor = 2 (selected via ablation in Table 3c)
    Controls the minimum distance between sampled prompt candidates; larger values reduce prompt count.
  • Prompt Elimination Threshold multiplier = Not explicitly stated; Table 3d suggests around 1.4
    Applied on top of the T_elim computation in Eq. 3; the ablation in Table 3d shows strong sensitivity, with mIoU ranging from 68.4 at 1.25 to 72.2 at 1.4.
assumptions (4)
  • domain assumption SA-1B point-prompt annotations are a valid supervision signal for 'essential' prompt locations in arbitrary images.
    The 'Training of Prompt Predictor' paragraph states that point prompts from SA-1B are used as ground truth. If this annotation distribution is not aligned with optimal prompts for other domains, the predicted confidence map is biased.
  • domain assumption SAM's image embedding, by itself, contains enough spatial and semantic information to localize useful point prompts.
    The Prompt Predictor takes only the raw image and the ViT embedding as inputs and is expected to output a dense confidence map; no external detection cues are used.
  • domain assumption A candidate point whose image feature is similar to the feature inside an already generated mask will produce a duplicate mask.
    ASF Eqs. 1 to 3 use cosine similarity between image features and reference mask features to eliminate prompts. The paper does not prove that feature similarity implies mask redundancy.
  • domain assumption Greedy IoU matching is a faithful accuracy metric for unlabeled SAM outputs.
    The evaluation uses the greedy IoU algorithm from Zhang et al. 2023d. The choice affects all compared methods equally, but it is not a standard metric like mAP and may reward certain mask-count distributions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AoP-SAM: Automation of Prompts for Efficient Segmentation." pith.science (2026). https://pith.science/paper/HSBIS4FH

@misc{pith2026250511980,
  author       = {Pith},
  title        = {Pith review of: AoP-SAM: Automation of Prompts for Efficient Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HSBIS4FH}},
  note         = {Machine review of arXiv:2505.11980}
}
read the original abstract

The Segment Anything Model (SAM) is a powerful foundation model for image segmentation, showing robust zero-shot generalization through prompt engineering. However, relying on manual prompts is impractical for real-world applications, particularly in scenarios where rapid prompt provision and resource efficiency are crucial. In this paper, we propose the Automation of Prompts for SAM (AoP-SAM), a novel approach that learns to generate essential prompts in optimal locations automatically. AoP-SAM enhances SAM's efficiency and usability by eliminating manual input, making it better suited for real-world tasks. Our approach employs a lightweight yet efficient Prompt Predictor model that detects key entities across images and identifies the optimal regions for placing prompt candidates. This method leverages SAM's image embeddings, preserving its zero-shot generalization capabilities without requiring fine-tuning. Additionally, we introduce a test-time instance-level Adaptive Sampling and Filtering mechanism that generates prompts in a coarse-to-fine manner. This notably enhances both prompt and mask generation efficiency by reducing computational overhead and minimizing redundant mask refinements. Evaluations of three datasets demonstrate that AoP-SAM substantially improves both prompt generation efficiency and mask generation accuracy, making SAM more effective for automated segmentation tasks.

Figures

Figures reproduced from arXiv: 2505.11980 by the authors.

Figure 1
Figure 1. In SAM, automating prompt provision eliminates the need for manual input, significantly improving the efficiency of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The architecture of our proposed AoP-SAM consists of two key components: the prompt predictor and the Adaptive [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 10 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al

    Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877--1901

  4. [4]

    Feng, D.; Haase-Sch \"u tz, C.; Rosenbaum, L.; Hertlein, H.; Glaeser, C.; Timm, F.; Wiesbeck, W.; and Dietmayer, K. 2020. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Transactions on Intelligent Transportation Systems, 22(3): 1341--1360

  5. [5]

    Gupta, A.; Dollar, P.; and Girshick, R. 2019. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 5356--5364

  6. [6]

    Han, D.; Zhang, C.; Qiao, Y.; Qamar, M.; Jung, Y.; Lee, S.; Bae, S.-H.; and Hong, C. S. 2023. Segment anything model (sam) meets glass: Mirror and transparent objects cannot be easily detected. arXiv preprint arXiv:2305.00278

  7. [7]

    Henderson, P.; and Ferrari, V. 2017. End-to-end training of object class detectors for mean average precision. In Computer Vision--ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part V 13, 198--213. Springer

  8. [8]

    H.; Jia, W.; He, X.; and Kennedy, P

    Hesamian, M. H.; Jia, W.; He, X.; and Kennedy, P. 2019. Deep learning techniques for medical image segmentation: achievements and challenges. Journal of digital imaging, 32: 582--596

Show all 25 references
  1. [9]

    Hu, J.; Lin, J.; Gong, S.; and Cai, W. 2024. Relax Image-Specific Prompt Requirement in SAM: A Single Generic Prompt for Segmenting Camouflaged Objects. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 12511--12518

  2. [10]

    Hu, J.; Tuo, H.; Wang, C.; Qiao, L.; Zhong, H.; and Jing, Z. 2019. Multi-Weight Partial Domain Adaptation. In BMVC, 5

  3. [11]

    Hu, J.; Tuo, H.; Wang, C.; Qiao, L.; Zhong, H.; Yan, J.; Jing, Z.; and Leung, H. 2020. Discriminative partial domain adversarial network. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XXVII 16, 632--648. Springer

  4. [12]

    V.; and Valada, A

    Hurtado, J. V.; and Valada, A. 2022. Semantic scene segmentation for robotics. In Deep learning for robot perception and cognition, 279--311. Elsevier

  5. [13]

    Kirillov, A.; He, K.; Girshick, R.; Rother, C.; and Doll \'a r, P. 2019. Panoptic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9404--9413

  6. [14]

    C.; Lo, W.-Y.; Doll \'a r, P.; and Girshick, R

    Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A. C.; Lo, W.-Y.; Doll \'a r, P.; and Girshick, R. 2023. Segment Anything. arXiv:2304.02643

  7. [15]

    Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Doll \'a r, P.; and Zitnick, C. L. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 1...

  8. [16]

    J.; Micorek, J.; Possegger, H.; and Bischof, H

    Mirza, M. J.; Micorek, J.; Possegger, H.; and Bischof, H. 2022. The norm must go on: Dynamic unsupervised domain adaptation by normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 14765--14775

  9. [17]

    H.; Sutawika, L.; Alyafeai, Z.; Chaffin, A.; Stiegler, A.; Scao, T

    Sanh, V.; Webson, A.; Raffel, C.; Bach, S. H.; Sutawika, L.; Alyafeai, Z.; Chaffin, A.; Stiegler, A.; Scao, T. L.; Raja, A.; et al. 2021. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207

  10. [18]

    Shotton, J.; Winn, J.; Rother, C.; and Criminisi, A. 2006. TextonBoost: Joint Appearance, Shape and Context Modeling for Mulit-Class Object Recognition and Segmentation. In European Conference on Computer Vision (ECCV)

  11. [19]

    Wang, D.; Shelhamer, E.; Liu, S.; Olshausen, B.; and Darrell, T. 2020. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726

  12. [20]

    Wang, G.; Chen, Y.; An, P.; Hong, H.; Hu, J.; and Huang, T. 2023 a . UAV-YOLOv8: A small-object-detection model based on improved YOLOv8 for UAV aerial photography scenarios. Sensors, 23(16): 7190

  13. [21]

    Wang, W.; Zhong, Z.; Wang, W.; Chen, X.; Ling, C.; Wang, B.; and Sebe, N. 2023 b . Dynamically instance-guided adaptation: A backward-free approach for test-time domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

  14. [23]

    U.; Bae, S.-H.; Lee, S.; and Hong, C

    Zhang, C.; Han, D.; Qiao, Y.; Kim, J. U.; Bae, S.-H.; Lee, S.; and Hong, C. S. 2023 b . Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289

  15. [24]

    Zhang, C.; Han, D.; Zheng, S.; Choi, J.; Kim, T.-H.; and Hong, C. S. 2023 c . Mobilesamv2: Faster segment anything to everything. arXiv preprint arXiv:2312.09579

  16. [25]

    D.; Zheng, S.; Li, C.; Qiao, Y.; Kang, T.; Shan, X.; Zhang, C.; Qin, C.; Rameau, F.; et al

    Zhang, C.; Puspitasari, F. D.; Zheng, S.; Li, C.; Qiao, Y.; Kang, T.; Shan, X.; Zhang, C.; Qin, C.; Rameau, F.; et al. 2023 d . A survey on segment anything model (sam): Vision foundation model meets prompt engineering. arXiv preprint arXiv:2306.06211

  17. [26]

    Zhang, R.; Jiang, Z.; Guo, Z.; Yan, S.; Pan, J.; Dong, H.; Gao, P.; and Li, H. 2023 e . Personalize Segment Anything Model with One Shot. arXiv preprint arXiv:2305.03048

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.