REVIEW 3 major objections 5 minor 23 references
Context Matters: Support Set Selection and Failure Detection for In-Context Medical Image Segmentation
T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This paper shows that in in-context medical image segmentation, retrieving support examples most similar to the query consistently matches or outperforms random sampling, and that a transformer trained on query and support embeddings can…
desk verdict Clean empirical study with a real new failure-detection component, but the similarity-selection gains may be inflated by image-level splits that leak same-patient frames into the support pool. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the support set itself, the only task-specific signal an in-context segmentation model receives. For the selection result, the machinery is cosine-similarity retrieval: query and support images are embedded with a frozen DINOv3 encoder, and the K support images closest to the query in embedding space form the support set. For the failure-detection result, the machinery is a four-layer transformer classifier: query and support embeddings are input tokens with learned query/support role embeddings added, support tokens are given no positional encoding so the classifier is invariant to support ordering, and a learned [CLS] token's output passes through an MLP to output a binary failure label (IoU below a dataset-specific threshold).
What would settle it
Measure the failure classifier's AUROC on similarity-selected support sets instead of the random ones it was trained on; if the AUROC drops to chance on any of the four benchmarks, the proposed screen-then-select deployment pattern collapses.
Extended reading notes
Core claim
The paper's central discovery is that the reliability of in-context segmentation is governed more by how the support set is chosen than by its size alone, and that this reliability is partially foreseeable before the mask is generated. Using MultiverSeg as the segmentation model and DINOv3 embeddings for image representation, the authors show that selecting the K support images with the highest cosine similarity to the query embedding gives higher median IoU and tighter distributions than random draws at every support-set size, with the advantage most pronounced at K=1 and K=2 (median IoU gains of 0.16 on EchoNet and 0.14 on WBC). They further train a transformer-based classifier that takes query and support embeddings, plus a learned class token, and predicts whether IoU will fall below the dataset's median threshold; it achieves AUROC between 0.69 and 0.80, exceeding chance on every benchmark. The authors interpret these results as evidence that both informed support selection and pre-use failure screening are practical levers for safer in-context segmentation.
Load-bearing premise
The proposed deployment pipeline first selects support sets by similarity and then screens them with a failure classifier, but the classifier was trained only on randomly sampled support sets, so its accuracy may not hold for similarity-selected inputs.
Editorial extensions
If this is right
- At small support-set sizes (K=1,2), similarity-based selection raises median IoU by 0.14–0.16 over random draws, so in the low-annotation regime where in-context learning is most attractive, informed selection is the largest lever.
- The gap between similarity and random sampling narrows as K grows and essentially closes by K=16, implying that retrieval quality matters most when the support set is smallest.
- Segmentation failure is predictable from embeddings alone, with AUROC up to 0.80 (EchoNet) and generally improving with K on three of four benchmarks, meaning a lightweight pre-screening step could flag unreliable outputs before deployment.
- Because the failure classifier needs only the query and support images—not the predicted mask—it can be applied before segmentation runs, enabling a two-stage pipeline of retrieve-then-screen.
Reading between the lines
- A direct test the paper does not perform: apply the trained failure classifier to similarity-selected support sets and measure AUROC; if performance drops, the classifier would need retraining or calibration on the deployment distribution.
- The permutation-invariant transformer design suggests the classifier encodes per-exemplar quality signals, so a natural extension is to rank candidate support sets by predicted success rather than merely flag failure.
- Since only one segmentation model (MultiverSeg) and one encoder (DINOv3) were used, the gains may be specific to that pairing; testing a second in-context segmentation model would show whether the selection benefit generalizes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies two mechanisms for improving the reliability of in-context medical image segmentation with MultiverSeg. It compares random support-set sampling with similarity-based retrieval using frozen DINOv3 embeddings on four benchmarks (EchoNet, WBC, HipXRay-Femur, HipXRay-Pelvis) for K = 1, 2, 4, 8, 16, 32, and it trains a transformer classifier to predict, from query and support image embeddings, whether the resulting segmentation IoU will fall below a dataset-specific median threshold. The authors report that similarity-based selection consistently matches or outperforms random sampling, with the largest gains at small support sizes (e.g., +0.16 median IoU at K = 1 on EchoNet), and that the failure classifier achieves AUROC between 0.69 and 0.80 across datasets. They conclude that informed support selection and pre-deployment failure screening are practical levers for safer clinical use.
Significance. The work addresses a genuinely important operational question for in-context segmentation: how to choose the support set and when to trust the output. The main strengths are that the selection mechanism is driven by frozen embeddings independent of the segmentation model (avoiding circularity in the evaluation), the random baseline is averaged over 40 draws per query, and the experimental protocol is clearly described. If the results are robust, the similarity-selection finding is directly actionable because it requires no retraining. The failure-detection result is also useful, although the absolute AUROC values are moderate and the proposed deployment pattern is not evaluated end-to-end. The present evidence is not yet strong enough for the word 'consistently' because of the single data split and the potential patient/video identity leakage.
major comments (3)
- [Section 3.1, Figure 3] The split into query and support pools is performed at the image level, not at the level of patients or videos. For EchoNet, the dataset provides two annotated frames per video (end-systole and end-diastole), so the same video can contribute one frame to the query pool and another to the support pool. Since similarity retrieval uses DINOv3 embeddings, the top support for a query under the similarity strategy will often be the paired frame from the same video, which is a near-duplicate of the query rather than an independent exemplar. The reported median-IoU gains at K = 1 and K = 2 in Section 3.2 may therefore be inflated by patient-identity leakage rather than by genuinely informative exemplars. Please re-run the selection comparison with a patient/video-level split, or explicitly exclude same-video supports, and report the fraction of retrieved supports that come from the same acquisition. The same risk should be checked for WBC and HipXRay if multiple images per patient or slide exist. This is the most load-bearing unverified assumption in the paper.
- [Section 3.3 and Section 4] The deployment pattern proposed in the Discussion first selects support sets by similarity and then screens them with the failure classifier, but the classifier is trained and evaluated entirely on support sets constructed by random sampling (Section 3.3, Eq. (1), Table 1). The distribution of query–support pairs under similarity selection is different from the random-sampling distribution, so the reported AUROC values may not transfer to the proposed pipeline. Please test the actual deployment pattern: evaluate the failure classifier on similarity-selected support sets, or at least report how performance changes when the random-trained classifier is applied to similarity-selected inputs.
- [Section 3.1, Section 3.2] All experiments use a single fixed 20:80 partition and a single random seed, and the central claims are made without confidence intervals or significance tests. The abstract's 'consistently matches or outperforms' and the similar statement in Section 3.2 are stronger than what a single split can establish, especially because Figure 3 shows overlap between the random and similarity distributions on some datasets and support sizes. Please provide results across multiple splits or bootstrap confidence intervals for the median-IoU differences and for the failure-detection AUROCs.
minor comments (5)
- [Section 2.2, Figure 4] Please specify how the transformer classifier handles variable K (e.g., padding and masking) and whether the classifier is trained jointly across all support sizes or separately per K; the text says results are pooled across support set sizes, but Figure 4 reports per-K AUROC.
- [Section 2.1, Figure 3 caption] The random baseline is reported as a per-query average over 40 trials, while the similarity strategy is a single deterministic draw; this asymmetry should be stated in the caption, because the box plots compare a distribution of means to a distribution of single draws.
- [Section 3.1] For EchoNet, please specify how the 'line segment annotations' were converted to binary segmentation masks (e.g., closed-contour filling), since the definition of IoU depends on the mask generation procedure.
- [Table 1, Section 3.3] The claim that AUROC is 'above chance' should be backed by confidence intervals or a statistical test; Figure 4 shows that HipXRay-Pelvis performance declines toward chance at large K, so the pooled AUROC of 0.71 should be accompanied by uncertainty estimates.
- [Section 3.1] Please report the number of distinct patients/videos per dataset and the number of images per patient/video, so readers can assess the leakage risk directly; the current description gives only image-level counts implicitly.
Circularity Check
No significant circularity: support selection and failure detection are empirically grounded with independent embeddings and standard supervised labels.
full rationale
The paper's derivation chain is self-contained against external benchmarks and does not reduce to its own inputs. Similarity-based support selection uses a frozen DINOv3 encoder g that is independent of the segmentation model f; the selected support set is the top-K cosine-similarity images, and no parameter is fitted to the IoU values being reported. The random-sampling baseline is measured over N=40 independent draws, and the comparison is an empirical evaluation rather than a construction. The failure-detection component is standard supervised learning: labels are defined in Eq. (1) by the model's actual IoU relative to a per-dataset median threshold τ, and the transformer classifier is trained on query/support embeddings with held-out query-level splits. The median-IoU threshold is a data-dependent label definition, not a fitted parameter later renamed as a prediction. The cited prior works (MultiverSeg, DINOv3, visual prompt selection papers) are external and do not overlap with the present authors, so no self-citation is load-bearing. The reader/skeptic concerns about image-level splits and the train/deployment distribution mismatch for similarity-selected supports are validity and generalization risks, not circularity, because they do not make any reported quantity equal to an input by construction. No step in the paper requires invoking the target result as an assumption.
Assumptions & free parameters
free parameters (1)
- per-dataset IoU failure threshold tau =
0.70 (EchoNet), 0.87 (WBC), 0.92 (HipXRay-Femur), 0.86 (HipXRay-Pelvis)
assumptions (3)
- domain assumption DINOv3 embedding cosine similarity is a valid proxy for how useful a support image will be for MultiverSeg segmentation of the query.
- domain assumption MultiverSeg and DINOv3 are representative of in-context segmentation models and image encoders respectively.
- domain assumption The single fixed 20:80 query/support split and 75:25 train/test split are representative of the data distribution.
Cite this review
Pith. "Pith review of Context Matters: Support Set Selection and Failure Detection for In-Context Medical Image Segmentation." pith.science (2026). https://pith.science/paper/HUEKIQSX
@misc{pith2026260805333,
author = {Pith},
title = {Pith review of: Context Matters: Support Set Selection and Failure Detection for In-Context Medical Image Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/HUEKIQSX}},
note = {Machine review of arXiv:2608.05333}
}
read the original abstract
In-context learning (ICL) adapts medical image segmentation models to unseen structures and modalities without retraining by conditioning on a task-specific support set of image-mask exemplars. Because this support set is the model's only task-specific signal, its composition directly influences segmentation performance. In this work, we investigate the support set as a controllable determinant of ICL reliability. First, we compare random sampling against similarity-based selection, where exemplars are retrieved based on their visual similarity to the query image. Second, we train a transformer-based classifier to predict, from the query and support images alone, whether a segmentation will fall below a specified Intersection-over-Union (IoU) threshold. Using MultiverSeg with DINOv3 embeddings across four benchmarks and three imaging modalities, we show that similarity-based selection consistently matches or outperforms random sampling, with the largest gains at the smallest support set sizes. Furthermore, our classifier predicts segmentation failure above chance on all four benchmarks. Ultimately, these results demonstrate that the reliability of in-context segmentation can be both improved via informed support selection and anticipated before use, providing practical mechanisms for safer clinical deployment.
Figures
Reference graph
Works this paper leans on
-
[1]
Advances in neural information processing systems33, 1877–1901 (2020)
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems33, 1877–1901 (2020)
2020
-
[2]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Butoi, V.I., Ortiz, J.J.G., Ma, T., Sabuncu, M.R., Guttag, J., Dalca, A.V.: Uni- verseg: Universal medical image segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 21438–21451 (2023)
2023
-
[3]
Nature medicine 24(9), 1342–1350 (2018)
De Fauw, J., Ledsam, J.R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Black- well, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al.: Clinically ap- plicable deep learning for diagnosis and referral in retinal disease. Nature medicine 24(9), 1342–1350 (2018)
work page 2018
-
[4]
In: Proceedings of the 2024 conference on empirical methods in natural language processing
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., et al.: A survey on in-context learning. In: Proceedings of the 2024 conference on empirical methods in natural language processing. pp. 1107–1128 (2024)
2024
-
[5]
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Un- terthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recogni- tion at scale. In: International Conference on Learning Representations (2021), https://openreview.net/forum?id=YicbFdNTTy
work page 2021
-
[6]
IEEE Transactions on Big Data (2025)
Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.E., Lomeli, M., Hosseini, L., Jégou, H.: The faiss library. IEEE Transactions on Big Data (2025)
2025
-
[7]
In: Proceed- ings of the Computer Vision and Pattern Recognition Conference
Gao, Y., Liu, D., Li, Z., Li, Y., Chen, D., Zhou, M., Metaxas, D.N.: Show and seg- ment: Universal medical image segmentation via in-context learning. In: Proceed- ings of the Computer Vision and Pattern Recognition Conference. pp. 20830–20840 (2025)
work page 2025
-
[8]
Nature biotechnology40(4), 555–565 (2022)
Greenwald, N.F., Miller, G., Moen, E., Kong, A., Kagel, A., Dougherty, T., Full- away, C.C., McIntosh, B.J., Leow, K.X., Schwartz, M.S., et al.: Whole-cell seg- mentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nature biotechnology40(4), 555–565 (2022)
2022
Show all 23 references
-
[9]
https://doi.org/10.17632/zm6bxzhmfz.1
Gut, D.: X-ray images of the hip joints (2021). https://doi.org/10.17632/zm6bxzhmfz.1
2021 doi
-
[10]
Clinical Oncology34(2), 74–88 (2022)
Harrison, K., Pullen, H., Welsh, C., Oktay, O., Alvarez-Valle, J., Jena, R.: Machine learning for auto-segmentation in radiotherapy planning. Clinical Oncology34(2), 74–88 (2022)
2022
-
[11]
Nature methods18(2), 203–211 (2021)
Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods18(2), 203–211 (2021)
2021
-
[12]
arXiv preprint arXiv:1412.6980 (2014) 10 Gehad et al
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014) 10 Gehad et al
2014 arXiv
-
[13]
Nature580(7802), 252–256 (2020)
Ouyang, D., He, B., Ghorbani, A., Yuan, N., Ebinger, J., Langlotz, C.P., Heiden- reich, P.A., Harrington, R.A., Liang, D.H., Ashley, E.A., et al.: Video-based ai for beat-to-beat assessment of cardiac function. Nature580(7802), 252–256 (2020)
2020
-
[14]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Rakic, M., Wong, H.E., Ortiz, J.J.G., Cimini, B.A., Guttag, J.V., Dalca, A.V.: Ty- che: Stochastic in-context learning for medical image segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11159–11173 (2024)
2024
-
[15]
Neuro-oncology27(8), 1946–1971 (2025)
Shi, T., Kujawa, A., Linares, C., Vercauteren, T., Booth, T.C.: Automated lon- gitudinal treatment response assessment of brain tumors: A systematic review. Neuro-oncology27(8), 1946–1971 (2025)
2025
-
[16]
arXiv preprint arXiv:2508.10104 (2025)
Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khali- dov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., et al.: Dinov3. arXiv preprint arXiv:2508.10104 (2025)
2025 arXiv
-
[17]
IEEE Transactions on Image Processing (2025)
Sun, Y., Chen, Q., Wang, J., Wang, J., Li, Z.: Exploring effective factors for im- proving visual in-context learning. IEEE Transactions on Image Processing (2025)
2025
-
[18]
arXiv preprint arXiv:2407.10233 (2024)
Suo, W., Lai, L., Sun, M., Zhang, H., Wang, P., Zhang, Y.: Visual prompt selection for in-context learning segmentation. arXiv preprint arXiv:2407.10233 (2024)
2024 arXiv
-
[19]
Advances in neural information pro- cessing systems30(2017)
Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems30(2017)
2017
-
[20]
Radiology: Artificial Intelligence 5(5), e230024 (2023)
Wasserthal, J., Breit, H.C., Meyer, M.T., Pradella, M., Hinck, D., Sauter, A.W., Heye, T., Boll, D.T., Cyriac, J., Yang, S., et al.: Totalsegmentator: robust segmen- tation of 104 anatomic structures in ct images. Radiology: Artificial Intelligence 5(5), e230024 (2023)
2023
-
[21]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Wong, H.E., Ortiz, J.J.G., Guttag, J., Dalca, A.V.: Multiverseg: scalable inter- active segmentation of biomedical imaging datasets with in-context guidance. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 20966–20980 (2025)
2025
-
[22]
Zhang, Y., Zhou, K., Liu, Z.: What makes good examples for visual in-context learning? Advances in Neural Information Processing Systems36, 17773–17794 (2023)
2023
-
[23]
Micron 107, 55–71 (2018)
Zheng, X., Wang, Y., Wang, G., Liu, J.: Fast and robust segmen- tation of white blood cell images by self-supervised learning. Micron 107, 55–71 (2018). https://doi.org/https://doi.org/10.1016/j.micron.2018.01.010, https://www.sciencedirect.com/science/article/pii/S0968432817303037
2018 doi
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.