{"id":"9c84a9b2-7edb-44dc-b899-eb0f9ce5b0ce","arxiv_id":"2501.09138","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Using a handful of labeled slices as memory in SAM2's video-segmentation pipeline enables prompt-free, fine-tuning-free segmentation of 3D medical volumes.","lead":"FATE-SAM adapts SAM2 to segment 3D medical images by using a few labeled slices as memory, with no model fine-tuning and no manual prompts. The paper reports strong Dice scores across 11 tasks on five public CT and MRI datasets, with the largest gains on thin structures like knee cartilage.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on SAM2's video-trained memory modules transferring to cross-patient medical masks, but the paper neither validates this transfer nor specifies how Eqs. (3)-(6) map onto SAM2's actual memory-attention interface; without code this is irreproducible.","rationale":"The reader's weakest assumption (frozen video-domain memory modules transferring to medical masks) is correct and load-bearing; this stress-test sharpens it with two concrete specifics: there is no direct validation that the memory pathway uses mask content, and the paper's Eqs. (3)-(6) do not specify how SAM2's real memory-attention interface, which includes per-object pointers and positional encodings inside the Hiera encoder, is invoked. These gaps do not overturn the empirical claim, but they justify the CONDITIONAL verdict: the central mechanism is plausible yet not established by the preprint alone. The paper's multi-task evaluation and ablations are real supporting evidence, and no internal contradiction was found, so I would not move to REJECT. The shuffled-mask test would directly settle whether the support masks' anatomical information drives performance, and a faithful reimplementation would settle the architectural-fidelity concern.","tokens_in":18220,"tokens_out":8355,"duration_ms":99617,"concrete_test":"On SKI10 and ACDC, run the full pipeline with the support masks spatially shuffled (random locations, same foreground pixel area) while keeping support images, retrieval, and all other steps identical. If average Dice remains near the reported 79.72% and 83.30%, then the exact anatomical mask content is not load-bearing; if it drops substantially, the memory transfer is real, and an independent reimplementation of Eqs. (1)-(9) with official SAM2 weights should be run to confirm the memory-attention interface matches the paper's description.","verdict_should_be":"UNCHANGED","load_bearing_attack":"FATE-SAM's central claim rests on Section III-A-2/3: a frozen SAM2 Memory Encoder and Memory Attention are fed support masks from other patients plus the previous slice's predicted mask, and this is what adapts SAM2 to medical anatomy (Eqs. (3)-(6)). The load-bearing premise is that representations trained on natural video masks remain meaningful when the memory inputs are cross-patient CT/MRI masks with no temporal relationship to the test slice. The paper never tests this premise directly. Ablations show that retrieval metric and volumetric consistency change Dice, but they do not isolate whether the memory encoder uses the mask content: aggregate Dice could be driven by the support image embeddings and the mask decoder, with the memory pathway acting as little more than a learned cross-attention bias. The concern is sharpened by an architectural gap: in SAM2, memory conditioning is built into the Hiera image encoder and uses per-object memory tokens/pointers plus positional encodings, whereas Eqs. (5)-(6) describe a generic concatenation followed by cross-attention and do not say how support masks from different volumes and object sets are packaged into SAM2's memory format. Without code and with no significance tests on Table III, the central reassembly mechanism is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FATE-SAM, a training-free and prompt-free method for 3D medical image segmentation built from frozen SAM2 components. The pipeline retrieves support slices by cosine similarity in Hiera image-embedding space, encodes support masks and the previous slice's predicted mask through SAM2's Memory Encoder, fuses these into unified memory embeddings, applies Memory Attention to the test-slice embedding, and decodes with SAM2's Mask Decoder. The method is evaluated on five public datasets covering 11 segmentation tasks, with ablations on SKI10 for support-set size, number of support examples, similarity metric, prompt type, volumetric consistency, initial slice, and SAM2 weight variant. The authors report the highest total average Dice (65.10%) among compared methods and particularly large gains on knee cartilage segmentation, while also acknowledging computational cost and difficulty with some small structures such as pancreatic cancer.","tokens_in":18476,"tokens_out":5590,"duration_ms":59505,"significance":"If the central claim holds, FATE-SAM is a practically valuable contribution: it demonstrates that frozen, video-pretrained SAM2 memory modules can be reassembled to perform few-shot medical segmentation without gradient updates or manual prompts, and it evaluates this across a broad set of CT and MRI tasks with public benchmarks. The strongest parts of the paper are the breadth of the evaluation, the explicit use of a held-out support set with frozen weights, and the ablation evidence that the volumetric-consistency mechanism contributes substantially (Table VI, average Dice 70.06 to 79.72 on SKI10). However, the significance is tempered by three gaps: the mapping from the described equations to SAM2's real memory interface is not specified in enough detail for reproduction, no statistical significance testing accompanies the numerical comparisons, and the exact preprocessing and implementation details are absent. These issues are fixable but currently leave the central empirical claim less secure than the text suggests.","major_comments":[{"comment":"The description of how the frozen SAM2 memory modules are invoked is underspecified and load-bearing. In SAM2, memory conditioning is integrated into the Hiera image encoder through memory tokens with specific positional encodings, and the Memory Encoder consumes a particular fusion of image embeddings and downsampled masks. Equations (3)-(6) instead describe generic elementwise additions and concatenation followed by self-/cross-attention, which does not establish that the official SAM2 Memory Encoder and Memory Attention are being used as claimed. The paper also does not state how the j support masks from different volumes and possibly different objects are packaged into memory tokens, nor how the per-object processing mentioned in Section III-A-3 interacts with the fused memory in Eq. (5). Since the paper's novelty is precisely the reassembly of SAM2 modules, the authors should provide exact tensor shapes, the official module names, positional-encoding handling, and either code or precise pseudocode that maps Eqs. (3)-(6) onto the actual SAM2 implementation.","section":"Section III-A-2/3, Eqs. (3)-(6)"},{"comment":"The empirical claim of 'consistently outperforming or matching competitors' is not supported by statistical evidence. Dice scores are reported as mean ± standard deviation over objects, not over volumes or repeated runs; several rows have only one or two objects, so the standard deviation is not a meaningful measure of segmentation variability. There are no significance tests, confidence intervals, or per-volume paired comparisons. Moreover, the per-task results are not uniformly favorable: on BTCV FATE-SAM averages 67.60 versus SAM-Med3D's 78.22, on BraTS 2017 it averages 41.58 versus MedSAM's 46.96, and on MSD Spleen it averages 84.21 versus SAM-Med3D's 94.17. The authors should either add paired statistical tests (for example, Wilcoxon signed-rank or bootstrap over volumes) and report per-volume variability, or substantially soften the consistency claim in the abstract and Section IV.A.","section":"Table III and Section IV.A"},{"comment":"Reproducibility is compromised by the absence of exact preprocessing and implementation details. The authors report only that experiments were run on Rocky Linux 8.8 with an NVIDIA A100 GPU and that SAM2 large weights were frozen. They do not describe how each 3D volume was converted into slices for SAM2's image encoder, what image resolution or spacing was used, how intensity windowing or normalization was applied, whether slices were resampled or padded, how the support set was constructed per task, or how the forward/backward propagation from the initial slice is terminated at volume boundaries. Medical segmentation results are highly sensitive to these choices, and Table III cannot be reproduced without them. The paper should include a complete preprocessing and inference protocol, and preferably release code.","section":"Section III.B.4 and Section IV"},{"comment":"The selection of key hyperparameters on SKI10 creates a circularity for the SKI10 result. The text states that j=3 support examples, cosine similarity, the central initial slice, and SAM2 large weights were chosen based on SKI10 ablations, and Section IV.A then reports SKI10 as a headline result (average Dice 79.72). Under these settings, the SKI10 number is not an independent evaluation of the configuration. The authors should explicitly acknowledge this in the limitations or use a nested or separate validation split so that the configuration is not selected and evaluated on the same dataset.","section":"Section III.B.3 and Section IV.A"},{"comment":"The formal notation for support retrieval and memory fusion is internally inconsistent. In Eq. (1), F_s is described as a library of support embeddings, but the dot product f_i · F_s is only defined for vectors, not for a set or library. In Eq. (2), the argmax ranges over both f_s^{ij} and y_s^{ij} with a similarity that is a function of embeddings only, so the mask should not appear in the optimization variable. More substantively, Eq. (5) concatenates one anatomical memory and one volumetric memory, yet the method selects j support examples (with j=3 in the final configuration); how multiple anatomical memories are pooled or ordered before concatenation is never specified. These ambiguities matter because the fusion of support information is the core of the proposed adaptation mechanism.","section":"Eqs. (1)-(2), (5)"}],"minor_comments":[{"comment":"There is a missing citation placeholder in the sentence about self-supervised learning techniques ('[?]'), and the sentence is incomplete as printed.","section":"Section II-A"},{"comment":"The text states that FATE-SAM outperforms 'fine-tuned SAMs like MedSAM2 (67.59%)', but the value 67.59% in Table III is the ACDC average for MedSAM, not MedSAM2 (whose average is 63.19%).","section":"Section IV.A, ACDC paragraph"},{"comment":"The row for SAM2 and the row for MedSAM2 both cite reference [33], which is 'Medical SAM 2: Segment medical images as video via segment anything model 2.' This conflates the base model and the fine-tuned medical variant; the authors should clarify which weights and checkpoints were used for each entry.","section":"Table II and References"},{"comment":"Minor typos in figure text: 'Swim-UNETR' appears in Figure 4 instead of 'Swin-UNETR', and the caption of Figure 3 says '8 competitive methods' while Table III lists nine other methods (UNet, UNETR, Swin-UNETR, SAM, SAM2, MedSAM, MedSAM2, SAM-Med3D, and Ours).","section":"Figure 3 and Figure 4"},{"comment":"The paper does not report inference runtime or memory usage, despite the discussion noting that the inference process can be computationally intensive; a quantitative runtime comparison would help readers assess the practical trade-off.","section":"Section III.B.4"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely and the empirical results are suggestive, but the manuscript is not yet reproducible: the SAM2 assembly is described too abstractly, preprocessing is absent, and no code is provided. In addition, the absence of significance tests makes the headline 'superior or comparable' claim hard to evaluate. I would be willing to reconsider after the authors supply the missing implementation details (ideally code), add statistical testing or per-volume variability, and address the SKI10 selection circularity. The paper may be better positioned as a journal article after these revisions; as is, it is not ready for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate, well-scoped empirical paper with a genuinely new pipeline, but the load-bearing mechanism is described at the wrong level of abstraction, and without code the results can't be checked.\n\nWhat's new: FATE-SAM reassembles frozen SAM2 components — image encoder, memory encoder, memory attention, mask decoder — to do prompt-free 3D medical segmentation from a small support set. The retrieval step (cosine similarity in Hiera feature space) and the volumetric consistency from the previous slice's prediction are both new in this combination. No gradient updates. The evaluation is broad: 11 tasks across five public CT/MRI datasets, versus supervised, zero-shot, and fine-tuned SAM baselines. The reported total average Dice (65.10) is the best in Table III, and the cartilage gains on SKI10 are large (66.81 vs MedSAM2's 26.69 for femoral cartilage). That is a real result, if it holds.\n\nWhere it gets soft: the paper is light on implementation details. Eqs. (3)-(6) present the memory fusion as 'concatenate then cross-attend,' but SAM2's memory attention uses per-object memory tokens with positional encodings, and it isn't explained how cross-patient support masks from different volume/object sets are packaged into that format. That is exactly the part that makes the method work, and it is underspecified. The stress-test note is right about this. There are also no significance tests, standard deviations are computed over objects rather than volumes, and no code is released. The ablations that set j=3, cosine similarity, central initial slice, and SAM2-large were all done on SKI10, which is also one of the test sets; that is mild circularity, not fatal, but it should be disclosed more clearly. The limitation section mentions compute cost and small structures but doesn't address the memory-interface gap.\n\nWould I trust the central claim? The empirical pattern — especially the volumetric consistency ablation raising cartilage Dice from 46 to 67 — is consistent with the memory pathway adding something real, not just acting as a bias. But the paper doesn't isolate whether the encoder uses mask content or just image embeddings, so I'd want code or a more direct analysis before relying on the numbers.\n\nBottom line: this is a serious paper for people working on SAM adaptation and few-shot medical segmentation. It deserves a proper referee — with code and better statistics it could be a solid contribution. I wouldn't cite it yet in my own work until the artifacts appear.","headline":"Genuinely new training-free SAM2 adaptation with broad results, but the memory mechanism is underspecified and the numbers need code and significance tests before I'd rely on them.","tokens_in":19024,"tokens_out":2324,"would_cite":false,"duration_ms":23547,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frozen SAM2 memory modules can segment 3D medical volumes from a handful of annotated slices, with no training and no manual prompts.","keywords":["training-free adaptation","few-shot segmentation","SAM2","medical image segmentation","volumetric consistency","prompt-free segmentation","support set retrieval","3D medical imaging"],"falsifier":"Run FATE-SAM on a fixed dataset and replace the retrieved support masks with anatomically incorrect or random masks while keeping the support images and all other settings unchanged; a large drop in Dice would confirm the anatomical memory is doing the work, whereas little or no drop would show the paper's stated mechanism is not needed. A complementary check is to compare cosine-similarity retrieval with random support selection; if outcomes match, the retrieval step is not load-bearing.","tokens_in":18019,"feed_emoji":"🩻","tokens_out":8831,"duration_ms":79556,"temperature":0.7,"pith_summary":"The paper introduces FATE-SAM, a method for segmenting 3D medical volumes that adapts SAM2 without any fine-tuning and without manual prompts. The core proposal is to treat SAM2's video-memory mechanics as a few-shot learner: a small support set of annotated slices from other patients is retrieved by image-feature similarity, encoded into memory by SAM2's frozen memory encoder, and used as anatomical guidance, while the prediction from the adjacent slice is fed back as volumetric memory. On eleven segmentation tasks spanning CT and MRI, FATE-SAM with 10% of volumes and three support examples per slice reports Dice scores that match or exceed fine-tuned medical SAMs on several datasets and beat supervised baselines trained on the same limited data. If the claim holds, clinical segmentation no longer requires large annotated datasets or expert-provided prompts for each new case.","feed_headline":"A 3-slice support set lets frozen SAM2 segment 3D medical scans","feed_subtitle":"FATE-SAM repurposes SAM2's video memory to carry anatomy across CT and MRI, matching fine-tuned baselines with no training.","key_machinery":"The load-bearing object is SAM2's memory system, specifically the memory encoder and memory attention module. In SAM2 these are designed for video frames; FATE-SAM reuses them to compress two kinds of guidance into a unified memory: retrieved cross-patient support examples ('anatomical memory') and the adjacent slice's prediction ('volumetric memory'). The support-retrieval step is a second key ingredient: it ranks support slices by cosine similarity between the frozen image encoder's embeddings and the test slice's embedding, selecting the top $j$ examples that enter the memory. The mask decoder then reads out a mask from the memory-enriched test embedding, and because the same memory encoder and attention are frozen, the entire adaptation is achieved by data selection and memory composition rather than weight updates.","core_discovery":"FATE-SAM's central claim is that SAM2's pre-trained memory modules can carry anatomy without being retrained. A test slice is encoded by the frozen image encoder; the top-ranked support slices are selected by cosine similarity in the Hiera feature space; support image embeddings and masks are passed through the memory encoder to form anatomical memory; the previous slice's image embedding and predicted mask are passed through the same encoder to form volumetric memory; the two are concatenated and attended to by the memory-attention module, and the mask decoder outputs the segmentation. The process propagates forward and backward from an initial slice, so each new prediction becomes memory for the next slice. The paper reports that with three support examples per slice and a support set of 10% of the volumes, FATE-SAM achieves average Dice scores of 79.72% on SKI10, 83.30% on ACDC, and 65.10% overall across all eleven tasks.","pith_inferences":["Because no weights are updated, the same frozen SAM2 checkpoint could serve many organs and modalities; adding a new anatomy would reduce to annotating a few slices rather than running a training job.","A sharper test than the paper's ablations would measure whether the volumetric-memory benefit decays with distance from the initial slice; if it does, the method is mostly slice-wise support retrieval rather than a true 3D prior.","The mechanism is not obviously limited to medicine: any dense-prediction task in which a few reference masks define the target category could in principle be solved by feeding those references through a frozen video-memory pipeline, so FATE-SAM is a candidate template for in-context segmentation generally."],"forward_implications":["On SKI10 knee MRI, FATE-SAM reaches 79.72% average Dice, above the supervised Swin-UNETR's 71.39% and MedSAM2's 62.44%, with the largest gains on thin cartilage structures.","On ACDC cardiac MRI, FATE-SAM reaches 83.30% average Dice, above U-Net's 73.95% and close to or above the fine-tuned medical SAMs, including 79.00% on the challenging myocardium.","Across all eleven tasks, FATE-SAM's 65.10% overall average Dice is higher than the supervised baselines (U-Net 44.93%, UNETR 34.78%, Swin-UNETR 46.62%) and the zero-shot SAM and SAM2 baselines, while remaining below the strongest fine-tuned medical SAM on the hardest abdominal task (BTCV: 67.60% vs SAM-Med3D's 78.22%).","Removing the volumetric-consistency memory drops SKI10 average Dice from 79.72% to 70.06%, and the ablation shows the gain concentrates on small cartilage structures, so the adjacent-slice memory is doing measurable work."],"supporting_citations":[{"why":"Supplies the frozen image encoder, memory encoder, memory attention, and mask decoder that FATE-SAM rewires into a few-shot segmenter.","marker":"[5]"},{"why":"MedSAM serves as the large-scale fine-tuned medical SAM baseline and the annotation-hungry paradigm FATE-SAM is designed to avoid.","marker":"[1]"},{"why":"SAM provides the zero-shot-with-manual-prompts baseline that motivates prompt-free adaptation.","marker":"[4]"},{"why":"SAM-Med3D is the fine-tuned volumetric medical SAM that outperforms FATE-SAM on the hardest abdominal task, marking the bar set by training.","marker":"[8]"},{"why":"Hiera is the MAE-pretrained image encoder in SAM2 whose frozen features are used in support retrieval and for all image embeddings.","marker":"[30]"},{"why":"MedSAM2 is the dataset-specific fine-tuned SAM2 baseline that treats volumes as video, the closest trained competitor to FATE-SAM's design.","marker":"[33]"},{"why":"The Medical Segmentation Decathlon supplies seven of the eleven evaluation tasks, grounding the claimed generalization across CT and MRI anatomy.","marker":"[42]"}],"fun_headline_variants":["Frozen SAM2 segments 3D medical scans with just 3 support slices","Three support slices let frozen SAM2 segment 3D medical volumes","No fine-tuning: FATE-SAM adapts SAM2 to medical imaging with few examples","FATE-SAM: few-shot 3D medical segmentation with frozen SAM2","Zero training, zero prompts: SAM2 segments 3D medical scans via 3 slices"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that SAM2's memory encoder and attention, trained on natural videos, treat masks from other patients' CT/MRI slices and the previous slice's predicted mask as valid memory inputs for medical anatomy despite the domain shift; if those frozen modules do not transfer, the few-shot gain the paper reports would disappear.","fun_headline_variants_meta":{"raw":{"variants":["Frozen SAM2 segments 3D medical scans with just 3 support slices","Three support slices let frozen SAM2 segment 3D medical volumes","No fine-tuning: FATE-SAM adapts SAM2 to medical imaging with few examples","FATE-SAM: few-shot 3D medical segmentation with frozen SAM2","Zero training, zero prompts: SAM2 segments 3D medical scans via 3 slices"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000956,"raw_usage":{"total_tokens":4112,"prompt_tokens":1017,"completion_tokens":3095,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":2988}},"tokens_in":633,"tokens_out":3095,"duration_ms":22179,"temperature":1.0,"reasoning_tokens":2988,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:10:01.863623+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FATE-SAM on a fixed dataset and replace the retrieved support masks with anatomically incorrect or random masks while keeping the support images and all other settings unchanged; a large drop in Dice would confirm the anatomical memory is doing the work, whereas little or no drop would show the paper's stated mechanism is not needed. A complementary check is to compare cosine-similarity retrieval with random support selection; if outcomes match, the retrieval step is not load-bearing.","supporting_citations":[{"cited_title":"Hiera: A hierarchi- cal vision transformer without the bells-and-whistles,","cited_arxiv_id":null,"evidence_quote":"Hiera is the MAE-pretrained image encoder in SAM2 whose frozen features are used in support retrieval and for all image embeddings."},{"cited_title":"The medical segmentation decathlon,","cited_arxiv_id":null,"evidence_quote":"The Medical Segmentation Decathlon supplies seven of the eleven evaluation tasks, grounding the claimed generalization across CT and MRI anatomy."}],"review_version":1}