Zero-Shot Gaze-based Volumetric Medical Image Segmentation

Ilya Pershin; Leila Khaertdinova; Tatyana Shmykova

REVIEW 4 major objections 7 minor 10 references

Zero-Shot Gaze-based Volumetric Medical Image Segmentation

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that gaze-based prompts can replace bounding boxes as a faster, slightly less accurate prompt modality for zero-shot 3D medical image segmentation.

desk verdict Gaze-prompting for 3D segmentation is a plausible but under-supported claim; the paper needs a timing protocol, multiple subjects, and statistical testing before the speed-accuracy tradeoff is convincing. read the letter →

arxiv 2505.15256 v2 pith:ZHHZIIBB submitted 2025-05-21 cs.CV cs.AI

Tatyana Shmykova , Leila Khaertdinova , Ilya Pershin This is my paper · ORCID

classification cs.CVcs.AI

keywords gaze-basedpromptinginteractivesegmentation3DmedicalimageSAM-2Medzero-shoteyetrackingabdominalCT

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces eye gaze as a prompt modality for interactive 3D medical image segmentation, claiming it is a time-efficient alternative to bounding boxes with a modest accuracy penalty. This works zero-shot with pre-trained SAM-2 and MedSAM-2 models, requiring no fine-tuning. On real data from one proxy radiologist on the WORD dataset, MedSAM-2 segmentation time fell from 88 to 62 seconds per volume while Dice fell from 0.844 to 0.759; synthetic gaze prompts showed the same direction. If this tradeoff holds, gaze could make volumetric annotation and interactive reading faster and more natural.

What carries the argument

The load-bearing mechanism is a gaze heatmap pipeline. Eye-tracking data is turned into 2D heatmaps via a Gaussian filter; K-Means clustering (the same post-processing as MedCLIP-SAMv2) converts each heatmap into a coarse mask; bounding boxes enclosing the connected components of that mask are passed as prompts to SAM-2 or MedSAM-2 with frozen weights. Shape-based interpolation between boolean masks propagates segmentation across the remaining CT slices, and an 80/20 in/out synthetic gaze-sampling scheme simulates fixation noise.

What would settle it

Run the same pipeline with five or more practicing radiologists on multiple eye-tracker workstations and CT datasets; the claim fails if gaze prompting no longer saves a practically meaningful amount of time (for example, well under 26 seconds on average) or if the Dice drop from bounding boxes exceeds about 0.1 in that multi-reader setting.

Watch

Extended reading notes

Core claim

The central claim is that gaze can serve as a viable prompt modality for zero-shot 3D medical image segmentation. Compared with bounding boxes, gaze-based prompts reduce interaction time at a modest cost in Dice score. On real data from a proxy radiologist, MedSAM-2 with gaze prompts segmented a CT volume in 62 seconds versus 88 seconds for bounding boxes, with Dice 0.759 versus 0.844; SAM-2 showed the same pattern (63 versus 88 seconds, Dice 0.750 versus 0.834). Synthetic gaze prompts on the WORD dataset were also faster (82 versus 102 seconds) and only slightly less accurate (0.817 versus 0.904 with MedSAM-2), while prompting only 30 of all slices preserved accuracy at reduced time. The authors further report that MedCLIP-SAMv2 text prompts performed poorly on WORD (Dice 0.16–0.20), which motivates gaze as an alternative zero-shot interaction that does not require fine-tuning.

Load-bearing premise

The load-bearing premise is that one trained proxy radiologist's gaze, captured with one eye tracker on one workstation, is representative of how real radiologists would produce gaze prompts in clinical practice.

Editorial extensions

If this is right

A radiologist using MedSAM-2 can segment an abdominal organ volume in about 62 seconds with gaze prompts instead of 88 seconds with bounding boxes, accepting a Dice drop from 0.844 to 0.759.
Prompting on 30 slices instead of every slice keeps accuracy essentially unchanged (Dice 0.904 versus 0.896–0.904 for bounding boxes) while reducing segmentation time from 133 to 102 seconds.
Gaze prompting requires no fine-tuning: the zero-shot pipeline works with frozen SAM-2 and MedSAM-2 weights, so it can be applied to any pre-trained SAM-family model.
The 80/20 synthetic gaze-noise scheme produces the same direction of results as real gaze, supporting the claim that the speed-accuracy tradeoff is not an artifact of one reader.
Text-prompt zero-shot segmentation is unreliable for narrow-domain targets (MedCLIP-SAMv2 gives Dice 0.16–0.20 on WORD), making gaze a plausible non-text alternative for such cases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Our inference: capturing gaze during normal image reading, rather than during a dedicated annotation session, could make the real time saving larger than the reported 26 seconds, because the radiologist is already looking at the volume for diagnosis.
Our inference: a hybrid protocol that uses gaze for initial localization and one or two corrective clicks on hard slices could recover most of the Dice gap (0.844 to 0.759) while preserving most of the time saving; the paper does not test this combination.
Our inference: because the pipeline converts gaze to boxes and feeds frozen SAM-family weights, the same approach should transfer to other foundation segmentation models and non-medical volumetric data, including video object segmentation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short take: The paper's central claim—gaze prompts are faster than bounding boxes with only a modest accuracy penalty—is plausible but not yet established by the reported numbers. The timing protocol is underspecified, and the single-subject evaluation is too thin to support the abstract's generalization.

What's new: this is the first evaluation of gaze-based prompts for 3D volumetric segmentation with SAM-2 and MedSAM-2. The pipeline—gaze heatmaps, K-Means clustering, box construction—is reused from prior 2D work and MedCLIP-SAMv2, but extending it to 3D with a memory-based model is a legitimate step. The paper does a decent job of comparing synthetic and real gaze data and reports the expected direction: gaze is faster, bbox is more accurate.

The soft spots are real. First, the time comparison in Table 3 lacks a protocol. Tables 1–2 show that time depends strongly on the number of prompted slices (all vs 30 vs first only), so without knowing how many slices were prompted in each condition, the 88 vs 62 s gap could be an artifact of prompt count rather than input modality. The paper also does not say whether the gaze time includes the heatmap post-processing and box-construction steps; if it excludes those, the comparison is not end-to-end. Second, the evidence base is one proxy radiologist, one workstation, one eye tracker. The paper itself concedes this in the Conclusion: 'the experiments with a proxy radiologist may not fully reflect real-world clinical conditions.' That is the load-bearing premise for the real-data tradeoff. Third, there is no significance testing or confidence intervals; the Dice drop from 0.844 to 0.759 is about 0.085 points, which is not obviously 'slightly lower' for clinical segmentation. The synthetic-data experiments are a useful sanity check, but the noise model (80/20 in/out) is arbitrary.

The citation pattern is fine: the self-citations are to prior gaze work, and the comparison is against external frozen models and the WORD dataset.

Who this is for: researchers working on interaction modalities for medical image segmentation, or on gaze-assistive annotation tools. They will get a clearly described pipeline and a reasonable first data point, but they should not treat the speed-accuracy tradeoff as established.

Recommendation: deserving of peer review, but a serious referee should push for a detailed timing protocol, a sensitivity analysis of the prompting slice count, and multiple subjects or statistical testing. With those, the core claim could be made solid; as it stands, it is conditional.

Referee Report

4 major / 7 minor

Summary. This paper proposes using continuous eye gaze as a prompt modality for zero-shot interactive segmentation of 3D abdominal CT volumes with SAM-2 and MedSAM-2. The proposed pipeline tracks gaze with a Tobii 4C eye tracker, builds Gaussian heatmaps, applies K-Means post-processing to derive bounding boxes, and feeds these boxes to frozen SAM-2/MedSAM-2; masks for unprompted slices are obtained by shape-based interpolation. The authors evaluate on the WORD dataset with synthetic gaze points sampled from ground-truth masks (80% inside, 20% outside the organ) and with real gaze from one trained proxy radiologist, comparing Dice scores and interaction time against bounding-box prompts. They report that gaze prompts reduce segmentation time from 88 to 62 seconds for MedSAM-2 while decreasing Dice from 0.844 to 0.759, and conclude that gaze offers a faster interaction with slightly lower segmentation quality.

Significance. The question addressed is practically relevant: annotation time in volumetric medical segmentation is a bottleneck, and gaze is a plausible low-effort prompt modality. The paper's strengths are that it uses two strong publicly available segmenters (SAM-2, MedSAM-2) in a zero-shot manner, evaluates on a public dataset (WORD), does not fine-tune the segmentation models, and explicitly acknowledges the proxy-radiologist limitation in the Conclusion. If the speed-accuracy tradeoff were rigorously established, this would be a useful contribution to interactive segmentation. However, the evidence currently rests on a single user and on an unreported timing protocol, so the significance is preliminary rather than demonstrated.

major comments (4)

[§4.3, Table 3] The central claim that gaze prompts are more time-efficient than bounding boxes is not established because the paper never specifies how the 88 s and 62 s times in Table 3 were measured. Tables 1–2 show that segmentation time depends strongly on the number of prompted slices (133 s vs. 102 s for all vs. 30 slices in Table 1; 108 s vs. 82 s in Table 2), so the reported 26 s gap could be an artifact if the bounding-box condition prompted more slices than the gaze condition or if the gaze time excludes the heatmap post-processing and box-construction steps (steps 2–4 of Figure 1). The authors should report the timing protocol, the number of prompted slices per condition, whether both conditions used identical slice subsets, and whether the reported times are end-to-end interaction times.
[§4.3, §4.1] The real-data comparison is based on a single trained proxy radiologist, with no repeated sessions, no inter-rater measurements, and no statistical significance testing or confidence intervals. Given the reported standard deviations (e.g., 0.844 ± 0.115 vs. 0.759 ± 0.203 for MedSAM-2 in Table 3), the claim of 'slightly lower' quality and 'time-efficient' is not supported without a paired test or at least effect-size quantification. The authors should add per-slice paired comparisons, confidence intervals on the time difference, and ideally multiple sessions or raters.
[§4.1, §4.3] The synthetic gaze model is an unvalidated assumption rather than a demonstrated proxy: points sampled from ground-truth masks with 80% inside/20% outside do not replicate real eye-tracking behavior, and the paper provides no comparison between synthetic and real gaze distributions. The gap between Table 2 (synthetic gaze Dice 0.809–0.817) and Table 3 (real gaze Dice 0.750–0.759) suggests the synthetic condition overestimates prompt quality, so the synthetic results should not be used as primary evidence for the gaze-based approach. Please report a validation of the synthetic noise model against the recorded real gaze data, or explicitly restrict the claims to the real-data experiment.
[§4.3] The text following Table 3 compares values that do not appear in Table 3 (0.904 vs. 0.896 and 82 vs. 102 secs are from Tables 1–2), and the paragraph mixes synthetic and real experiments without clearly separating them. Since the central comparison in the abstract is the real-data tradeoff, the authors should reorganize this section so that Table 3's numbers are the focus and synthetic results are clearly labeled as auxiliary.

minor comments (7)

[Abstract] The abstract says 'slightly lower segmentation quality' without quantifying the drop; a decrease from 0.844 to 0.759 in Dice may not be perceived as slight by clinical users, so the claim should be qualified with the actual numbers.
[§2] The sentence beginning 'Unlikely, multimodal models' should read 'Unlike multimodal models'; the description of the MedCLIP-SAMv2 evaluation on the WORD dataset would also benefit from reporting the exact prompt settings and the number of organs tested.
[§3] The heatmap post-processing parameters (Gaussian smoothing sigma, K-Means cluster count, contour threshold) are not given, so the pipeline is not reproducible from the text alone.
[§4.1] The paper does not report how the 350 CT slices are distributed across the 16 organs or how many slices per organ were used in the real gaze experiment, which makes it difficult to assess whether the reported Dice and time averages are balanced across anatomical structures.
[Tables 1–2] Table 1's caption says 'on synthetic data' even though the prompts are generated from ground-truth mask boundaries rather than from gaze, while Table 2 uses 'synthetic gaze heatmaps'; using the same word 'synthetic' for both conditions makes the two tables easy to conflate.
[§5] The conclusion contains grammar and wording issues: 'gaze-based prompting offer' should be 'gaze-based prompting offers', and the term 'masklet' appears to be a typo for 'mask'.
[General] No code, trained models, or processed gaze data are provided, which limits the reproducibility of the real-data experiment and the synthetic noise model.

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical comparison against external models and dataset; self-citations supply gaze processing and noise inputs but do not encode the measured speed-accuracy tradeoff.

full rationale

The paper's central claim is empirical: gaze-based prompts are faster than bounding boxes with slightly lower Dice. This is supported by measured comparisons using the externally developed SAM-2 and MedSAM-2 models on the WORD dataset. The only author self-citations, [6] and [7], are used for gaze heatmap post-processing and for the synthetic gaze noise model (80% inside, 20% outside the organ); neither of these inputs fixes the speed or accuracy comparison reported in Table 3. The real-prompt times and Dice scores are measured outcomes, not consequences of those cited gaze-processing choices. There is no equation in which a predicted quantity is defined in terms of the comparison target, no fitted parameter renamed as a prediction, and no load-bearing uniqueness theorem imported from the authors' prior work. The timing protocol is underspecified and the proxy-radiologist sample is a validity limitation, but these are experimental-design concerns rather than circularity. Under the stated rules, the derivation chain is self-contained relative to external benchmarks, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The listed items are hand-chosen modeling choices and domain assumptions; none of the free parameters is fitted to the reported Dice values, and the paper introduces no new physical or conceptual entities.

free parameters (3)

Gaussian heatmap smoothing sigma
Gaze heatmap generation in Section 3 uses a Gaussian filter; the kernel width is not reported and appears taken from MedCLIP-SAMv2 [4].
K-Means cluster count
Heatmap post-processing in Section 3 applies K-Means clustering to form a coarse mask; the number of clusters is not specified.
Synthetic gaze noise split = 80% inside / 20% outside
Section 4.1 samples 80% of synthetic gaze points inside the organ and 20% outside to model tracker error; this ratio is chosen, not measured.

assumptions (4)

domain assumption Pre-trained SAM-2 and MedSAM-2 produce valid segmentation masks from box prompts on CT images without fine-tuning.
Section 4 evaluates only prompt construction; the quality and behavior of the frozen backbones are taken as given.
domain assumption The MedCLIP-SAMv2 post-processing pipeline (K-Means clustering on saliency maps) transfers to gaze heatmaps.
Section 3 states 'We employed the same post-processing method as MedCLIP-SAMv2 [4]' without a dedicated validation of this transfer.
domain assumption One trained proxy radiologist's gaze is representative enough to estimate clinical performance.
Section 4.3 bases real-data results on a single proxy radiologist; the Conclusion flags this limitation.
ad hoc to paper Synthetic gaze noise, modeled as 80% points inside and 20% outside the organ, approximates real eye-tracking error.
Section 4.1 introduces this split with a citation to [7], but the ratio is a modeling choice not derived from measured tracker noise.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Zero-Shot Gaze-based Volumetric Medical Image Segmentation." pith.science (2026). https://pith.science/paper/ZHHZIIBB

@misc{pith2026250515256,
  author       = {Pith},
  title        = {Pith review of: Zero-Shot Gaze-based Volumetric Medical Image Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZHHZIIBB}},
  note         = {Machine review of arXiv:2505.15256}
}

read the original abstract

Accurate segmentation of anatomical structures in volumetric medical images is crucial for clinical applications, including disease monitoring and cancer treatment planning. Contemporary interactive segmentation models, such as Segment Anything Model 2 (SAM-2) and its medical variant (MedSAM-2), rely on manually provided prompts like bounding boxes and mouse clicks. In this study, we introduce eye gaze as a novel informational modality for interactive segmentation, marking the application of eye-tracking for 3D medical image segmentation. We evaluate the performance of using gaze-based prompts with SAM-2 and MedSAM-2 using both synthetic and real gaze data. Compared to bounding boxes, gaze-based prompts offer a time-efficient interaction approach with slightly lower segmentation quality. Our findings highlight the potential of using gaze as a complementary input modality for interactive 3D medical image segmentation.

Figures

Figures reproduced from arXiv: 2505.15256 by the authors.

**Figure 1.** The process of generating segmentation masks based on the annotator’s gaze stream involves the following steps: (1) tracking of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Gaze-based system that enables radiologists to segment [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 6 canonical work pages

[1]

Segment anything

Kirillov Alexander, Mintun Eric, Ravi Nikhila, Mao Hanzi, Rolland Chloe, Gustafson Laura, Xiao Tete, Whitehead Spencer, Berg Alexander C, Lo Wan-Yen, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023. 1

work page 2023
[2]

Segment anything in medical images

Ma Jun, He Yuting, Li Feifei, Han Lin, You Chenyu, and Wang Bo. Segment anything in medical images. Nature Communications, 15(1):654, 2024. 1

work page 2024
[3]

Medical sam 2: Segment medical images as video via segment anything model 2

Jiayuan Zhu Abdullah Hamdi Yunli Qi Yueming Jin Junde Wu. Medical sam 2: Segment medical images as video via segment anything model 2. arXiv preprint arXiv:2408.00874, 2024. 1

arXiv 2024
[4]

Medclip-samv2: Towards universal text- driven medical image segmentation

Taha Koleilat Hojat Asgariandehkordi Hassan Rivaz Yim- ing Xiao. Medclip-samv2: Towards universal text- driven medical image segmentation. arXiv preprint arXiv:2409.19483v3, 2024. 1, 3

arXiv 2024
[5]

Gazesam: What you see is what you segment.arXiv preprint arXiv:2304.13844, 2023

Bin Wang Armstrong Aboah Zheyuan Zhang Ulas Bagci. Gazesam: What you see is what you segment.arXiv preprint arXiv:2304.13844, 2023. 1

arXiv 2023
[6]

Gaze-Assisted Medical Image Segmentation

Leila Khaertdinova Ilya Pershin Tatiana Shmykova Bu- lat Ibragimov. Gaze-assisted medical image segmentation. arXiv preprint arXiv:2410.17920, 2024. 1

work page Pith review arXiv 2024
[7]

Gaze assistance for efficient segmentation correction of medical images

Khaertdinova Leila, Shmykova Tatyana, Pershin Ilya, Laryukov Andrey, Khanov Albert, Zidikhanov Damir, and Ibragimov Bulat. Gaze assistance for efficient segmentation correction of medical images. IEEE Access, 2025. 1, 3

work page 2025
[8]

Word: A large scale dataset benchmark and clinical applicable study for abdomi- nal organ segmentation from ct image

Luo Xiangde, Liao Wenjun, Xiao Jianghong, Chen Jieneng, Song Tao, Zhang Xiaofan, Li Kang, Metaxas Dimitris N, Wang Guotai, and Zhang Shaoting. Word: A large scale dataset benchmark and clinical applicable study for abdomi- nal organ segmentation from ct image. Medical Image Anal- ysis, 82:102642, 2022. 1, 3

work page 2022

Show all 10 references

[9]

Sam 2: Segment anything in images and videos

Nikhila Ravi Valentin Gabeur Yuan-Ting Hu Ronghang Hu Chaitanya Ryali Tengyu Ma Haitham Khedr Roman R ¨adle Chloe Rolland Laura Gustafson Eric Mintun Junting Pan Kalyan Vasudev Alwala Nicolas Carion Chao-Yuan Wu Ross Girshick Piotr Doll ´ar Christoph Feichtenhofer. Sam 2: Segm...

2024 arXiv
[10]

Ef- ficient semiautomatic segmentation of 3d objects in medical images

Schenk Andrea, Prause Guido, and Peitgen Heinz-Otto. Ef- ficient semiautomatic segmentation of 3d objects in medical images. In International Conference on Medical Image Com- puting and Computer-Assisted Intervention, pages 186–195. Springer, 2000. 3

2000

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Segment anything

Kirillov Alexander, Mintun Eric, Ravi Nikhila, Mao Hanzi, Rolland Chloe, Gustafson Laura, Xiao Tete, Whitehead Spencer, Berg Alexander C, Lo Wan-Yen, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023. 1

work page 2023

[2] [2]

Segment anything in medical images

Ma Jun, He Yuting, Li Feifei, Han Lin, You Chenyu, and Wang Bo. Segment anything in medical images. Nature Communications, 15(1):654, 2024. 1

work page 2024

[3] [3]

Medical sam 2: Segment medical images as video via segment anything model 2

Jiayuan Zhu Abdullah Hamdi Yunli Qi Yueming Jin Junde Wu. Medical sam 2: Segment medical images as video via segment anything model 2. arXiv preprint arXiv:2408.00874, 2024. 1

arXiv 2024

[4] [4]

Medclip-samv2: Towards universal text- driven medical image segmentation

Taha Koleilat Hojat Asgariandehkordi Hassan Rivaz Yim- ing Xiao. Medclip-samv2: Towards universal text- driven medical image segmentation. arXiv preprint arXiv:2409.19483v3, 2024. 1, 3

arXiv 2024

[5] [5]

Gazesam: What you see is what you segment.arXiv preprint arXiv:2304.13844, 2023

Bin Wang Armstrong Aboah Zheyuan Zhang Ulas Bagci. Gazesam: What you see is what you segment.arXiv preprint arXiv:2304.13844, 2023. 1

arXiv 2023

[6] [6]

Gaze-Assisted Medical Image Segmentation

Leila Khaertdinova Ilya Pershin Tatiana Shmykova Bu- lat Ibragimov. Gaze-assisted medical image segmentation. arXiv preprint arXiv:2410.17920, 2024. 1

work page Pith review arXiv 2024

[7] [7]

Gaze assistance for efficient segmentation correction of medical images

Khaertdinova Leila, Shmykova Tatyana, Pershin Ilya, Laryukov Andrey, Khanov Albert, Zidikhanov Damir, and Ibragimov Bulat. Gaze assistance for efficient segmentation correction of medical images. IEEE Access, 2025. 1, 3

work page 2025

[8] [8]

Word: A large scale dataset benchmark and clinical applicable study for abdomi- nal organ segmentation from ct image

Luo Xiangde, Liao Wenjun, Xiao Jianghong, Chen Jieneng, Song Tao, Zhang Xiaofan, Li Kang, Metaxas Dimitris N, Wang Guotai, and Zhang Shaoting. Word: A large scale dataset benchmark and clinical applicable study for abdomi- nal organ segmentation from ct image. Medical Image Anal- ysis, 82:102642, 2022. 1, 3

work page 2022