{"id":"11b89baf-d982-441a-ac55-cc51d19ae392","arxiv_id":"2412.13908","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A plug-in memorizing transformer retrieves training-time key-value features by kNN search and improves 3D SAM segmentation by 11.36 average Dice points on TotalSegmentator.","lead":"Memorizing SAM adds a memory block to a 3D medical segmentation model, letting it recall features from training scans during inference. The authors report an average Dice gain of 11.36% over FastSAM3D on 33 classes from TotalSegmentator, with only a few extra milliseconds added per image.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-class training confound: Section 2 says N separate class-specific models are trained, so the 11.36% gain over FastSAM3D may reflect specialization, not the memory plug-in.","rationale":"The reader identified memory-bank representativeness as the weakest assumption, but I find the more fundamental issue is the fairness of the comparison. Section 2 explicitly trains a separate model per class, while the baseline FastSAM3D appears to be a single multi-class model. The reported average Dice gain may therefore be attributable to per-class specialization rather than to the memorizing Transformer block. This concern is load-bearing because the central claim is that the memory mechanism is a plug-in that improves 3D SAM; if the improvement is actually due to training 33 specialized models, the memory component is not validated. The proposed test would settle this by adding the missing control. I still view the paper as plausible and not internally inconsistent, and the public code would help, but the current written methods omit enough detail to rule out the confound. The reader's conditional verdict remains appropriate, so I do not change the verdict; I only sharpen the condition needed for acceptance.","tokens_in":4638,"tokens_out":5627,"duration_ms":51951,"concrete_test":"Fine-tune FastSAM3D separately on each of the 33 one-class datasets, using the same data and training protocol as Memorizing SAM but without the memory branch, and compute the average Dice over the same 500 test volumes. If this per-class baseline already matches most of the reported 11.36% improvement, the memory module's marginal contribution is small. Also run Memorizing SAM with memory key-value pairs retrieved from a different class or with shuffled values; if Dice does not drop substantially, the retrieval content is not doing the work.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is not the representativeness of the memory bank but the experimental comparison described in Section 2. The methods state: 'we train N separate Memorizing SAM models, each specialized for one of the N object classes,' where N=33. The headline result compares the average Dice of these class-specialized Memorizing SAM models against FastSAM3D. If FastSAM3D is a single multi-class model, the comparison conflates two interventions: per-class specialization and memory-augmented attention. A per-class model only needs to separate one foreground from background, which is an easier task and could explain a large part of the 11.36% average Dice gain. The paper does not report a per-class fine-tuned FastSAM3D baseline without memory, nor does it describe how the 33 class-specific models are combined at inference or how inference time is aggregated. Without this control, the central causal claim that the memorizing block is responsible for the improvement is not established. This is a correctness risk, not a disagreement with consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Memorizing SAM, a 3D medical image segmentation model that augments FastSAM3D with a memorizing transformer block. During training, class-specific models are trained and their internal key-value representations are saved as external memory; at inference, kNN search retrieves top-k pairs that are combined with local attention via a ratio. Evaluated on 33 TotalSegmentator classes (200 training, 10 tuning, 500 test), the method is reported to outperform FastSAM3D by 11.36% average Dice with only 4.38 ms additional inference time. The authors provide code and claim no extra FLOPs or parameters.","tokens_in":4832,"tokens_out":3846,"duration_ms":32963,"significance":"If the reported gains hold under controlled comparison, the memorizing plug-in would be a low-cost way to improve 3D SAM without retraining, especially for under-represented classes. The idea of using precomputed high-quality representations as memory is elegant, and the code release is a strength. However, the current experiments do not isolate the effect of memory from per-class specialization, so the central claim is not yet established; the paper needs additional control experiments and corrected efficiency reporting.","major_comments":[{"comment":"The method trains N separate Memorizing SAM models, one per class (N=33), and the headline average Dice improvement is computed against FastSAM3D, which is presented as a single multi-class model. This confounds the memory plug-in with per-class specialization, since each specialized model solves an easier binary segmentation task. Without a per-class FastSAM3D baseline (fine-tuned without the memory block) or a multi-class Memorizing SAM, the 11.36% gain cannot be attributed to the memorizing mechanism.","section":"Section 2, Overview of Memorizing SAM"},{"comment":"The reported FLOPs are identical (23.14 G) for FastSAM3D and Memorizing SAM, despite Memorizing SAM adding kNN searches and k attention computations in the memorizing block. This is not physically consistent unless the kNN operations are excluded from the measurement; please report the actual FLOPs including the memory block or clarify precisely which operations are counted.","section":"Table 1"},{"comment":"All performance claims rest on single point estimates with no error bars, standard deviations, or significance tests, and the only baseline is FastSAM3D. Please report repeated-run variability and add at least one additional 3D SAM-based baseline; otherwise the robustness of the 11.36% improvement cannot be assessed.","section":"Section 3, Results"},{"comment":"The k hyperparameter is selected by ablation, but the text does not state whether the ablation is performed on the 10-sample tuning set or on the 500-sample test set; if test data are used for model selection, the reported Dice values are optimistically biased. Please specify the data split used for choosing k and RL, and avoid test-set-based hyperparameter selection.","section":"Section 3, Ablation Study"},{"comment":"The paper does not explain how the 33 class-specific models are run at inference time or how the 6.89 ms per-image inference time is aggregated across classes. If all 33 models must process each volume to produce the final multi-class segmentation, the effective inference time is much larger than 6.89 ms; this directly affects the claim of only 4.38 ms additional cost.","section":"Sections 2-3, inference protocol"}],"minor_comments":[{"comment":"The term 'un-fine-tuned' is used repeatedly; please replace with 'not fine-tuned' or 'zero-shot' for clarity.","section":"Throughout the text"},{"comment":"The phrase 'FastSAM3Di.e., FastSAM3D' is missing a space after 'FastSAM3D'; the same typo appears in the abstract and in Section 2.","section":"Abstract"},{"comment":"The value '11.36 ˙%' contains a stray dot and should be '11.36%'.","section":"Section 3, Results"},{"comment":"The equation for Ri is not typeset clearly ('DiPk i=0 RL Di'); please rewrite the combination rule with clear notation, define all symbols, and specify how the distances Di are normalized.","section":"Section 2, Memorizing Transformer Block"},{"comment":"The subfigures are extremely small and the caption is dense; the per-class Dice comparison would be more readable as a table or box plot with class names.","section":"Figure 2"},{"comment":"The paper does not describe the prompt setting used for all models; since SAM is prompt-based, specify whether the same prompts (e.g., bounding boxes) are provided to FastSAM3D and Memorizing SAM.","section":"Section 2, Overview of Memorizing SAM"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is short and appears to be an early version. The central idea is interesting, but the evaluation as written does not support the headline claim due to the per-class confound and unclear FLOPs reporting. If the authors add the controlled experiments requested, the paper could become suitable. I did not review the public code repository."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a reasonable engineering paper with a confounded headline number. The idea—bolt a kNN-attention memory block onto a 3D SAM and fill it with precomputed key-value pairs from high-quality internal representations—is not a new mechanism (it follows Memorizing Transformers and MoViT), but packaging it as a plug-in for FastSAM3D and testing on 33 TotalSegmentator classes is a useful thing to have. The design choice of preparing memory before inference and loading it on demand is practical, and the 4.38 ms/image overhead is attractive if it holds up.\n\nWhat the paper does not yet establish is where the 11.36% average Dice gain comes from. Section 2 says they train N separate Memorizing SAM models, one per class. The baseline FastSAM3D is described as a single model, but the paper never says whether it was also fine-tuned per class. If not, the comparison bundles two interventions: per-class specialization (a much easier binary problem) and the memory block. A per-class FastSAM3D without memory is the missing control, and without it the central causal claim is not supported. If the authors actually used per-class FastSAM3D, they need to say so; the text as written doesn't.\n\nOther soft spots, in decreasing order. There are no error bars or significance tests across the 33 classes, so we are looking at point estimates. Only one baseline is used, from the same group. Table 1 reports identical FLOPs after adding kNN attention, which is at minimum underspecified; if FLOPs excludes the lookup and the extra attention, that should be stated. The phrase 'particularly marginal improvements in un-fine-tuned scenarios' contradicts the later claim that memory is especially beneficial without fine-tuning; probably a typo, but it should be fixed. The k hyperparameter is selected using the separate 10-sample tuning set, so that particular concern is less serious than it first looks—but the paper should be explicit that test data was not used for model selection.\n\nOn citations, the paper cites the right prior work and evaluates against an external public benchmark, so the result is falsifiable once the code is released. Formal evidence is limited to the dataset and promised code; there is no verified artifact.\n\nWho is this for: applied medical imaging researchers who want a cheap retrofit to a SAM variant for volumetric segmentation. The method description is compact and reproducible in principle. I would send this to review, with the per-class baseline question as the main condition. If the gain survives that control, it is a solid incremental contribution. If it doesn't, the paper is still a useful description of a plug-in that may not be worth the complexity.","headline":"A useful plug-in idea whose headline gain is still confounded by per-class specialist training versus the single-model baseline.","tokens_in":5357,"tokens_out":5140,"would_cite":false,"duration_ms":42509,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a retrievable memory of high-quality training representations to a 3D SAM raises its average Dice by 11.36% at a cost of 4.38 ms per image.","keywords":["medical image segmentation","Segment Anything Model","Memorizing Transformer","3D volumetric segmentation","kNN attention","TotalSegmentator","foundation models"],"falsifier":"Run Memorizing SAM on a CT dataset from a different scanner vendor or patient population whose anatomical variants are absent from the 210 memorized volumes; if the average Dice against FastSAM3D falls to zero or negative, the memory-representativeness premise is false. A second check: replace the stored memory keys with random keys from the same volumes; if Dice does not drop substantially, the improvement comes from the kNN operation itself rather than the memory content.","tokens_in":4421,"feed_emoji":"🩻","tokens_out":8090,"duration_ms":57470,"temperature":0.7,"pith_summary":"This paper asks whether a memorizing Transformer—a plug-in that retrieves stored key-value representations of past inputs—can close the performance gap of 3D Segment Anything Models on volumetric medical segmentation. It introduces Memorizing SAM, which precomputes a memory bank from class-specific fine-tuned models on 210 training and tuning volumes, saves it to disk, and retrieves the top-k memories via kNN attention during inference. Evaluated on 33 TotalSegmentator classes and 500 test volumes, the method beats the FastSAM3D baseline by an average of 11.36% Dice while adding only 4.38 ms per image and no new parameters. The authors argue the benefit is largest when the SAM has not been task-specifically fine-tuned, which matters because well-annotated 3D medical data for fine-tuning is scarce.","feed_headline":"Memory plug-in lifts 3D medical SAM Dice by 11.36%","feed_subtitle":"Retrieving top-k memorized training volumes adds only 4.38 ms per image and helps most when fine-tuning data is scarce.","key_machinery":"The central object is the Memorizing Transformer block, a plug-in that retains the standard dense self-attention and feed-forward layers and adds a kNN attention branch. For a query, it retrieves the top-$k$ (key, value) pairs from a precomputed external memory and computes attention ratios $R_i$ from the kNN distances $D_i$, combining them with the local attention output $A_L$ as $A_c = A_L \\cdot R_L + \\sum_{i=0}^k R_i \\cdot A_i$, with the local ratio $R_L$ set to 0.3. The memory itself is generated offline: the dataset is split into $N$ one-class subsets, a class-specific SAM is fine-tuned on each, and the resulting internal representations are saved as (key, value, image) tuples to disk, loaded into cache only at retrieval time. This design keeps FLOPs at 23.14 G and parameters at 45.64 M, identical to the FastSAM3D baseline, while adding 4.38 ms per image for the retrieval and attention computation.","core_discovery":"The central claim is that a Memorizing Transformer block, used as a plug-in in 3D SAM, improves segmentation accuracy by retrieving and attending to precomputed internal representations of high-quality training examples. The memory is built by training one-class SAM models on a small dataset (200 fine-tuning samples), generating key-value pairs for each class, and storing them in external memory alongside the images. During inference, the block performs an approximate kNN search over these keys and combines the retrieved attention outputs with local attention using a ratio-based weighting scheme. Measured on 33 classes of TotalSegmentator, Memorizing SAM outperforms FastSAM3D by 11.36% average Dice, with inference time rising from 2.51 ms to 6.89 ms per image while FLOPs, parameters, and GPU memory stay essentially unchanged. The authors interpret the gains as evidence that memory retrieval can substitute for part of the fine-tuning signal, especially for un-fine-tuned models.","pith_inferences":["If the memory bank is representative, the same retrieval mechanism could be extended to other foundation models in medical imaging, for instance to adapt a model to a new hospital's scan protocol by storing a small set of labeled cases from that site.","A direct ablation would be to corrupt the memory content, e.g., by replacing stored keys with random or wrong-class representations; if Dice does not drop substantially, the gain comes from the kNN operation itself rather than the memorized content.","The results hint at a 'memorize instead of fine-tune' trade-off, where retrieving expert-annotated slices approximates part of the benefit of gradient updates—potentially relevant in federated or privacy-constrained settings where raw training data cannot be shared but representations might be.","Because the reported timing is on an NVIDIA T4 GPU, the 4.38 ms overhead may scale differently on edge or CPU hardware; measuring retrieval latency on deployment hardware would clarify clinical feasibility."],"forward_implications":["Memorizing SAM consistently beats FastSAM3D across most of the 33 anatomical classes, with the largest gains on challenging or underrepresented structures.","The memory mechanism is most valuable for models that have not been task-specifically fine-tuned, suggesting it can serve as a lightweight alternative to fine-tuning when annotated volumes are scarce.","The performance gain comes with negligible overhead: inference time increases by 4.38 ms per image, while FLOPs, memory usage, and parameter count stay essentially flat.","Varying the number of retrieved memories k has only a modest effect, with k=3 giving the best trade-off; later-ranked, less similar memories contribute little to the output.","Because the block is a plug-in and memory is precomputed, the approach can be attached to any SAM variant, not only FastSAM3D."],"supporting_citations":[{"why":"Supplies the base Segment Anything Model architecture whose Transformer block the memorizing plug-in extends.","marker":"[9]"},{"why":"The state-of-the-art 3D SAM baseline that Memorizing SAM builds on and is compared against for Dice, IoU, and efficiency.","marker":"[16]"},{"why":"Introduces the memorizing Transformer mechanism, including the ratio-based combination of local and kNN attention that the plug-in adapts.","marker":"[17]"},{"why":"Extends memorizing Transformers to vision and demonstrates their use for medical image analysis, motivating the plug-in design.","marker":"[18]"},{"why":"Provides the TotalSegmentator dataset, which supplies the 33 classes, the 200 fine-tuning, 10 tuning, and 500 test samples used in evaluation.","marker":"[19]"}],"fun_headline_variants":["Memory-augmented 3D SAM beats FastSAM3D by 11.4% Dice","kNN memory plug-in adds 4 ms, lifts 3D SAM Dice 11%","Retrieval memory helps 3D SAM most when fine-tuning data is scarce","Plug-in memory block lifts 3D medical SAM accuracy with minimal cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The precomputed memory bank, built from 210 training and tuning volumes, must contain key-value pairs representative enough that kNN retrieval helps on 500 unseen test volumes across 33 anatomical classes; if the stored keys fail to cover the anatomical variation in the test set, retrieval returns irrelevant values and the reported 11.36% Dice gain disappears.","fun_headline_variants_meta":{"raw":{"variants":["Memory-augmented 3D SAM beats FastSAM3D by 11.4% Dice","kNN memory plug-in adds 4 ms, lifts 3D SAM Dice 11%","Retrieval memory helps 3D SAM most when fine-tuning data is scarce","Plug-in memory block lifts 3D medical SAM accuracy with minimal cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00056,"raw_usage":{"total_tokens":2686,"prompt_tokens":999,"completion_tokens":1687,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":1595}},"tokens_in":615,"tokens_out":1687,"duration_ms":12158,"temperature":1.0,"reasoning_tokens":1595,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:39:56.332585+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Memorizing SAM on a CT dataset from a different scanner vendor or patient population whose anatomical variants are absent from the 210 memorized volumes; if the average Dice against FastSAM3D falls to zero or negative, the memory-representativeness premise is false. A second check: replace the stored memory keys with random keys from the same volumes; if Dice does not drop substantially, the improvement comes from the kNN operation itself rather than the memory content.","supporting_citations":[{"cited_title":"FastSAM3D: An Efficient Segment Anything Model for 3D Volumetric Medical Images","cited_arxiv_id":"2403.09827","evidence_quote":"The state-of-the-art 3D SAM baseline that Memorizing SAM builds on and is compared against for Dice, IoU, and efficiency."},{"cited_title":"Movit: Memorizing vision transformers for medical image analysis,","cited_arxiv_id":null,"evidence_quote":"Extends memorizing Transformers to vision and demonstrates their use for medical image analysis, motivating the plug-in design."},{"cited_title":"Totalsegmentator: Robust segmentation of 104 anatomic structures in ct images,","cited_arxiv_id":null,"evidence_quote":"Provides the TotalSegmentator dataset, which supplies the 33 classes, the 200 fine-tuning, 10 tuning, and 500 test samples used in evaluation."}],"review_version":1}