{"id":"fd6c32a4-9b57-4548-a8a0-07e2f593b34d","arxiv_id":"2504.13340","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"On meniscus segmentation from 3D knee MRI, end-to-end fine-tuned SAM matches 3D U-Net in Dice score but remains inferior in Hausdorff distance, while decoder-only SAM performs worse on both.","lead":"Researchers fine-tuned the Segment Anything Model on 3D knee MRI to segment the meniscus and compared it with a basic 3D U-Net. When only the decoder was fine-tuned, SAM scored a lower Dice of 0.81 versus 0.87 for U-Net; end-to-end fine-tuning closed that gap but SAM still matched the meniscus shape less accurately.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Morphology claim may hinge on unremoved SAM artifacts: connected-component post-processing could close the Hausdorff gap and weaken the unsuitability conclusion.","rationale":"The reader's weakest assumption was that slice-wise 2D processing is a fair way to apply SAM to 3D knee MRI, with connected-component post-processing listed as one possible alternative. My stress-test converges on the same area but sharpens it: the paper's own connected-component counts show that SAM outputs numerous small islands, and the 95th-percentile Hausdorff metric used to support the morphology claim is precisely the type of metric that such islands inflate. This makes the morphological inferiority conclusion potentially artifact-driven rather than a true geometric limitation. The Dice results, by contrast, are internally consistent and likely robust: SAM 1 at 0.81±0.03 clearly trails U-Net's 0.87±0.03, while SAM 2 matches it, consistent with the IWOAI winner. Thus the central quantitative comparison stands, but the broader suitability claim and the morphology-specific conclusion need a post-processing ablation before they can be taken at face value. This does not change the reader's conditional verdict; it strengthens the rationale for requiring the proposed check as a condition of acceptance. I partially agree with the reader because I emphasize the Hausdorff-artifact mechanism more than the general 3D-adaptation fairness, though both are part of the same underlying concern.","tokens_in":13012,"tokens_out":4985,"duration_ms":48137,"concrete_test":"Re-run the SAM 1 and SAM 2 test-set evaluations after applying connected-component post-processing that keeps only the largest K components (K=2 by default, matching the expected two menisci; sweep K=1..5 for robustness), then recompute Dice, 95% Hausdorff Distance, and average thickness. If SAM 2's Hausdorff becomes statistically indistinguishable from U-Net's, the paper's morphology-based unsuitability conclusion fails; if Hausdorff remains significantly worse after this standard post-processing, the conclusion is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 reports that SAM 2 predictions contain an average of 10.2 connected components versus 2.3 for 3D U-Net, and Figure 5 shows 'flecks of unattached positive predictions.' The morphology conclusion is based largely on the 95th-percentile Hausdorff Distance (Table 2: 2.4±1.4 mm for SAM 2 versus 1.8±0.8 mm for U-Net), a metric that is highly sensitive to exactly such spurious islands. No connected-component post-processing is applied to any model, while the text itself notes that U-Net 'did not suffer from this problem.' It is therefore possible that the reported Hausdorff gap reflects removable artifacts rather than a genuine inability to capture meniscus geometry. In addition, the broader statement that SAM 'may not be suitable for similar 3D medical image segmentation tasks' extrapolates from a single slice-wise 2D adaptation of SAM with anisotropic sagittal slices and no 3D context; this design does not test 3D-aware adaptations or volume-based prompting. The core Dice comparison is internally consistent, but the stronger morphological-suitability claim remains unsettled unless these artifacts are controlled.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates two fine-tuning configurations of the Segment Anything Model (SAM) against a 3D U-Net for automatic meniscus segmentation in 3D knee MRI from the IWOAI 2019 dataset. The authors report that decoder-only fine-tuning (SAM 1) yields a Dice score of 0.81±0.03, described as significantly worse than the U-Net's 0.87±0.03, while end-to-end fine-tuning (SAM 2) reaches 0.87±0.03, comparable to the U-Net and to the IWOAI winning score. On the 95th-percentile Hausdorff distance, both SAM variants are inferior to the U-Net, leading the authors to conclude that SAM may not be suitable for similar fine-structure, low-contrast 3D segmentation tasks. The experiments use consistent splits with the challenge, a held-out test set, and multiple metrics; code is provided.","tokens_in":13264,"tokens_out":3429,"duration_ms":29859,"significance":"If the findings hold, the paper makes a useful empirical contribution to the debate on foundation models for medical imaging: it shows that decoder-only fine-tuning is insufficient for low-contrast small structures, and that end-to-end fine-tuning can match a well-trained 3D U-Net on Dice but leaves a residual morphology gap. The study is clearly designed, uses a public dataset and challenge splits, and releases code. The main claims, however, rest on a small test set and on a Hausdorff comparison that may be affected by removable isolated predictions; also, the 'significantly worse' statement is not formally tested.","major_comments":[{"comment":"The claim that SAM 1 'performed significantly worse' than the 3D U-Net is not supported by a statistical test. With n=28 test volumes, the reported Dice difference (0.81±0.03 vs 0.87±0.03) should be accompanied by a paired test, bootstrap confidence interval, or a non-parametric alternative. The same applies to the Hausdorff-distance comparisons in Table 2. Without such an analysis, the reader cannot judge whether the observed differences are beyond sampling variation.","section":"Section 4 (Conclusion) and Abstract"},{"comment":"The Hausdorff-distance evidence for morphological inferiority of SAM is confounded by the isolated prediction components acknowledged in the text (10.2 connected components for SAM 2 versus 2.3 for the U-Net). The 95th-percentile Hausdorff distance is highly sensitive to exactly these spurious islands. Please apply a largest-connected-component filter or a small-component removal step to both models' outputs and re-report Dice and Hausdorff distances. If the gap persists, the morphology claim is strengthened; if it closes, the conclusion that SAM 'is inferior ... in preserving the spatial features of the menisci' would need to be revised or conditioned on the absence of post-processing.","section":"Section 3, Table 2, and Figure 5"},{"comment":"The generalization that SAM 'may not be suitable for similar 3D medical image segmentation tasks' is too broad for a study that tests only a slice-wise 2D adaptation of SAM. The paper does not compare against a 2D U-Net trained on the same slices, nor does it evaluate a 3D-aware adaptation or volume-based prompting. Adding a 2D U-Net baseline (or a 3D SAM variant) would isolate whether the observed gap is intrinsic to SAM or an artifact of the 2D/3D interface. Without this control, the conclusion should be tempered to the specific slice-wise adaptation evaluated here.","section":"Section 4 (Conclusion) and Section 2.4"}],"minor_comments":[{"comment":"The table contains formatting artifacts, such as '000...888777±±± 000...000333', which should be cleaned to read '0.87±0.03'.","section":"Table 2"},{"comment":"The winning IWOAI Dice score is reported as '0.88±0.3'; given the other reported standard deviations, this is almost certainly a typo for '0.88±0.03'.","section":"Section 3, first paragraph"},{"comment":"The word 'detatched' should be 'detached' in the sentence describing the separated medial meniscus.","section":"Section 3.1"},{"comment":"The description 'Three copies of this upsampled image were concatenated' is clear, but it would be helpful to state explicitly that this creates the required 3-channel RGB input for SAM.","section":"Section 2.4"},{"comment":"The average transverse thickness metric would benefit from a precise definition of 'non-zero columns in the transverse plane', including whether medial and lateral menisci are considered jointly or separately.","section":"Section 2.5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward empirical benchmark with a useful negative result about decoder-only fine-tuning of SAM. The core Dice comparison is internally consistent, but the two strongest claims (statistical significance and morphological unsuitability) are under-supported by the current analysis. The Hausdorff artifact concern is testable with simple post-processing, and the over-generalization to 'similar 3D tasks' should be addressed with a 2D baseline or softened wording. I recommend major revision rather than rejection because the central design and data are sound and the concerns are addressable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline: this is a clean, useful benchmark showing that decoder-only fine-tuning of SAM is insufficient for low-contrast 3D meniscus segmentation (Dice 0.81 vs 0.87 for U-Net), and even end-to-end fine-tuning only matches a basic 3D U-Net on Dice, not on morphology as measured by Hausdorff distance. The comparison is new and the setup is honest: consistent IWOAI 2019 splits, held-out test, code on GitHub, both SAM configurations and a U-Net baseline trained from scratch.\n\nThe paper does several things right. It exposes a real practical question – whether a 2D foundation model can be squeezed into a 3D medical task with minimal modification – and answers it without hand-waving. The case analyses of the torn meniscus and partial meniscectomy are informative. The reporting of parameters and grid search is transparent.\n\nThe soft spots are in proportion. The stress-test note has a point: the Hausdorff gap (2.4 vs 1.8 mm) is exactly the kind of metric that gets inflated by the spurious islands SAM produces (10.2 connected components vs 2.3 for U-Net). No connected-component removal was applied, and the text even notes U-Net didn't suffer from this. Without that post-processing or a lesion-level analysis, the 'SAM is inferior at preserving morphology' claim is not as settled as the conclusion suggests. The 'may not be suitable for similar 3D tasks' in the abstract overreaches from one slice-wise 2D adaptation; it should be qualified as 'this slice-wise adaptation.' Also, the claim that SAM 1 performed 'significantly worse' lacks a significance test. With n=28 and these standard deviations, a test would almost certainly confirm it, but the paper should report one rather than rely on the word 'significantly.' Minor: the preprocessing choices (cropping, windowing threshold) are heuristic and selected by looking at the data; the authors note this, and it's acceptable for a benchmark but worth flagging.\n\nDice-wise the central comparison is solid, and the conclusion that end-to-end fine-tuning matters more than the frozen encoder is well supported. The morphology conclusion is plausible but softer than the text admits.\n\nWho is this for? Anyone thinking of repurposing SAM for fine-grained 3D segmentation. It deserves a serious referee: the experiment is reproducible, the question is practical, and the weaknesses are fixable with modest effort (significance test, connected-component ablation, and toning down the generalization claim). I'd send it to review as is; it's not a desk reject.\n\nBest,","headline":"Useful benchmark showing decoder-only SAM fine-tuning is not enough for 3D meniscus segmentation, but the morphology claim needs an artifact-controlled check.","tokens_in":13802,"tokens_out":2246,"would_cite":false,"duration_ms":20306,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning only SAM's decoder is not enough to segment knee menisci; even full end-to-end training that matches 3D U-Net on Dice leaves SAM behind on morphology.","keywords":["meniscus segmentation","Segment Anything Model (SAM)","3D knee MRI","3D U-Net","fine-tuning","Dice score","Hausdorff distance","osteoarthritis"],"falsifier":"The claim would be settled by rerunning the same experiment on the same test volumes with one change: give SAM access to 3D context, either by volume-based prompting or by removing small disconnected components before scoring. If a 3D-aware SAM variant reaches a 95th-percentile Hausdorff distance at or below the 3D U-Net's 1.8 mm while keeping Dice at or above 0.87, the paper's conclusion would not survive; if the morphology gap persists under every reasonable 3D adaptation, the conclusion would be strengthened.","tokens_in":12792,"feed_emoji":"🦵","tokens_out":7413,"duration_ms":60004,"temperature":0.7,"pith_summary":"The paper tests whether the Segment Anything Model (SAM), a widely used 2D segmentation foundation model, can be adapted to fully automatic segmentation of knee menisci in 3D MRI. It finds that fine-tuning only SAM's mask decoder, the cheapest adaptation, yields a Dice score of 0.81±0.03 on held-out scans, significantly below the 0.87±0.03 of a basic 3D U-Net. Training all of SAM's parameters end-to-end raises the Dice score to match the U-Net (0.87±0.03), on par with the winning score in the 2019 knee MRI segmentation challenge, but SAM still has larger Hausdorff distances and produces fragmented masks. If this is right, SAM's generalisability does not carry over to fine 3D structures with low contrast and unclear boundaries, and a simple 3D U-Net remains the safer practical choice.","feed_headline":"SAM matches U-Net on meniscus Dice only after full fine-tuning","feed_subtitle":"Decoder-only tuning scores 0.81 vs 0.87; full tuning matches overlap but not shape.","key_machinery":"The central object is the Segment Anything Model in its base ViT configuration, treated as a 2D slice-wise segmenter: each 160-slice sagittal volume is split, each slice is bilinearly upsampled and padded to 1024×1024 and replicated to three channels, SAM runs without prompts and outputs one mask, and the slices are stacked back into a 3D volume. The comparison machinery is a 3D U-Net with 16 initial feature maps and three convolution blocks per encoder and decoder, trained from random weights. Two fine-tuning regimes isolate what carries the argument: a frozen image encoder with only the mask decoder trained (about 4 million parameters) versus full end-to-end training (about 94 million parameters). The metrics that carry the conclusions are Dice score for overlap, 95th-percentile Hausdorff distance for boundary morphology, and average transverse thickness difference as a biomarker-relevant quantity.","core_discovery":"The paper's central claim is that SAM, when adapted to 3D knee MRI by processing sagittal slices independently, can reach but not exceed the Dice-score performance of a standard 3D U-Net for meniscus segmentation, and only when it is fine-tuned end-to-end. Decoder-only fine-tuning is insufficient: frozen SAM features do not carry enough task-relevant information for low-contrast menisci, producing 0.81 Dice versus the U-Net's 0.87. End-to-end fine-tuning restores overlap performance (0.87), matching the 0.88 winning score of the 2019 challenge, yet the 95th-percentile Hausdorff distance remains worse (2.4±1.4 mm versus 1.8±0.8 mm), and SAM predictions contain far more disconnected components, indicating inferior preservation of meniscus morphology. The paper concludes that SAM fine-tuning may therefore be unsuitable for segmentation of similar fine anatomical structures and for deriving morphological biomarkers from such segmentations.","pith_inferences":["Going beyond the paper: a volume-aware adaptation of SAM, such as joint 3D prompting, multi-slice attention, or connected-component post-processing, might erase the reported Dice and Hausdorff gaps, and the slice-wise protocol is the natural first thing to vary.","Going beyond the paper: the same comparison could be run on other small low-contrast joint structures, for example the acetabular labrum or spinal discs, to see whether the pattern is general or specific to menisci.","Going beyond the paper: because the ground truths came from a single expert annotating slice-wise, part of U-Net's Hausdorff advantage may reflect matching staircase-style annotation rather than true anatomy; a multi-rater or smooth ground-truth benchmark would test whether SAM's textured output is actually closer to real tissue shape."],"forward_implications":["A 2D foundation model should not be assumed transferable to 3D medical segmentation of small, low-contrast structures; slice-wise application with a frozen encoder underperforms a basic 3D U-Net.","End-to-end fine-tuning of SAM can recover Dice-score parity with a 3D U-Net and with challenge-winning performance, making it a viable overlap-based baseline when full training is affordable.","If morphology matters, SAM's slice-wise masks are worse than U-Net's: the paper reports a mean Hausdorff distance of 2.4 mm versus 1.8 mm and an average of 10.2 versus 2.3 disconnected components, so SAM output would need more post-processing and could misrepresent geometry.","For meniscal thickness as a biomarker, both fully fine-tuned SAM and U-Net overestimate average thickness by sub-voxel amounts, so overlap-based parity does not automatically mean identical downstream biomarker estimates."],"supporting_citations":[{"why":"introduces SAM and its large-scale pretraining, which motivates the generalisability expectation being tested","marker":"[18]"},{"why":"introduces the U-Net architecture whose design the 3D baseline extends to volumetric convolution","marker":"[33]"},{"why":"provides the 3D U-Net formulation used as the state-of-the-art baseline","marker":"[6]"},{"why":"supplies the standardized knee MRI dataset, the train/validation/test splits, and the challenge-winning Dice score used as the performance reference","marker":"[9]"},{"why":"prior work fine-tuning SAM's mask decoder on medical images, which this study extends to a fully automatic, prompt-free meniscus task","marker":"[24]"},{"why":"reports SAM's weakness on ill-defined boundaries, which motivates the choice of menisci as a hard case","marker":"[16]"}],"fun_headline_variants":["SAM needs full fine-tuning to match U-Net on meniscus","Decoder-only SAM fails on knee MRI; full tuning ties U-Net","Meniscus MRI: SAM matches U-Net only after full fine-tuning","SAM's Dice ties U-Net, but shape loss worse on knee MRI","Full fine-tuning required for SAM to match U-Net on menisci"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that feeding SAM sagittal slices independently and stacking the resulting masks is a fair way to apply a 2D foundation model to a 3D knee MRI; if a 3D-aware adaptation would behave differently, the paper's conclusion about SAM's unsuitability would not follow.","fun_headline_variants_meta":{"raw":{"variants":["SAM needs full fine-tuning to match U-Net on meniscus","Decoder-only SAM fails on knee MRI; full tuning ties U-Net","Meniscus MRI: SAM matches U-Net only after full fine-tuning","SAM's Dice ties U-Net, but shape loss worse on knee MRI","Full fine-tuning required for SAM to match U-Net on menisci"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000684,"raw_usage":{"total_tokens":3209,"prompt_tokens":1154,"completion_tokens":2055,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":770,"completion_tokens_details":{"reasoning_tokens":1958}},"tokens_in":770,"tokens_out":2055,"duration_ms":12102,"temperature":1.0,"reasoning_tokens":1958,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:10:08.110637+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The claim would be settled by rerunning the same experiment on the same test volumes with one change: give SAM access to 3D context, either by volume-based prompting or by removing small disconnected components before scoring. If a 3D-aware SAM variant reaches a 95th-percentile Hausdorff distance at or below the 3D U-Net's 1.8 mm while keeping Dice at or above 0.87, the paper's conclusion would not survive; if the morphology gap persists under every reasonable 3D adaptation, the conclusion would be strengthened.","supporting_citations":[{"cited_title":"The International Workshop on Osteoarthritis Imaging Knee MRI Segmentation Challenge: A Multi-Institute Evaluation and Analysis Framework on a Standardized Dataset","cited_arxiv_id":"2004.14003","evidence_quote":"supplies the standardized knee MRI dataset, the train/validation/test splits, and the challenge-winning Dice score used as the performance reference"}],"review_version":1}