REVIEW 3 major objections 5 minor 30 references
Curia-MAE: Multi-Modal Multi-Anatomy MAE Pre-Training for 3D Medical Image Segmentation
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Curia-MAE claims that a single frozen 3D encoder, pre-trained on 300,000 CT and MRI volumes, can be reused across diverse segmentation tasks, improving over a strong MAE baseline and matching nnU-Net under full fine-tuning, with the…
desk verdict Careful engineering and honest ablations, but the headline frozen-encoder gain rests on single runs and could be noise; worth a serious referee if the authors add seed variance control. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a local–global similarity objective added on top of the masked autoencoder. A cropped, masked view of a volume and the full unmasked volume pass through the same encoder; the feature embedding of the local crop is aligned with the embedding of the matching region extracted from the complete volume, using a Barlow Twins redundancy-reduction criterion that drives the cross-correlation matrix toward the identity. This objective is what the ablations show producing the principal frozen-encoder gains, since it forces the encoder to predict consistent representations of the same anatomy across masking and scale. The other components — Huber loss for reconstruction, a context loss on unmasked patches, SIGReg isotropic feature regularization, and random spacing augmentation — support fine-tuning robustness and robustness to acquisition heterogeneity rather than delivering the headline frozen-encoder improvement.
What would settle it
Retrain Curia-MAE and the MAE baseline multiple times (for example, five seeds each) under the full 16-node pre-training schedule, then compare the frozen-encoder Dice gaps on the eight benchmarks (average anatomy 66.8 to 67.5, lesion 63.0 to 64.7) against the seed-to-seed standard deviation; if the gap is within noise, the central claim fails.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that the pre-training objective, not the architecture or dataset scale alone, is what makes a frozen encoder viable for dense 3D segmentation. Curia-MAE improves average frozen-encoder Dice over the MAE baseline on both anatomical (66.8 to 67.5) and lesion (63.0 to 64.7) benchmarks, with the largest single-task advance on TS-MRI (36.9 to 40.6); under full fine-tuning it matches the baseline on anatomy and exceeds nnU-Net on lesion tasks (69.4 vs 68.6). The authors attribute the frozen-encoder gains primarily to the local–global similarity objective with Barlow Twins, while the reconstruction-side components are kept for fine-tuning and acquisition robustness. The authors also report a modest average gain over the baseline on image-level classification and regression tasks, indicating that the dense pre-training does not sacrifice global representation quality.
Load-bearing premise
The load-bearing assumption is that one pre-training run per configuration measures the real difference between Curia-MAE and the baseline; the paper reports uncertainty by resampling test cases only, not by retraining the model, so the small average gains could vanish under run-to-run variance.
Editorial extensions
If this is right
- A single frozen encoder can be reused across diverse CT and MRI segmentation tasks by training only a small decoder per task, lowering adaptation cost.
- Lesion segmentation, where labeled data is scarce, benefits most from this pre-training, both in the frozen and fine-tuned regimes.
- The full-scale Curia-MAE improves frozen-encoder Dice over the MAE baseline on five of the eight benchmarks at the dataset level, with the clearest gains on AMOS-CT, TS-MRI, MSD-Lung, MSD-Pancreas, and Atlas.
- The method's components are architecture-agnostic in principle, so the recipe may transfer to transformer-based backbones.
- Public release of the pre-trained weights would allow other groups to reuse the encoder without access to the 300,000-volume pre-training set.
Reading between the lines
- If the frozen-encoder gains survive multi-seed retraining, the practical threshold for clinical deployment is a decoder-only adaptation pipeline that runs on a single GPU, which the paper's setup already approximates.
- The local–global similarity objective should transfer to other dense prediction targets such as registration or detection, because it enforces scale- and corruption-consistency rather than task-specific semantics.
- A direct test of the random-spacing component would be to evaluate all models at non-canonical voxel spacings; the paper keeps the evaluation at 1.0×1.0×1.0 mm, so the component's claimed robustness benefit remains untested in-distribution.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Curia-MAE, a self-supervised pre-training method for 3D medical image segmentation. It extends a convolutional MAE baseline with a Huber reconstruction loss, a context loss, a SIGReg feature regularizer, random spacing augmentation, and a local–global similarity objective instantiated with Barlow Twins. The model is pre-trained on about 300,000 CT and MRI volumes and evaluated on eight segmentation benchmarks under frozen-encoder and full-fine-tuning regimes, plus classification/regression tasks on CuriaBench3D. The central claim is that Curia-MAE improves frozen-encoder segmentation performance over a strong MAE baseline while remaining competitive under full fine-tuning and superior on lesion tasks.
Significance. If the central claim is established, the contribution is useful: it provides evidence that a single frozen convolutional encoder, pre-trained with masked autoencoding plus joint-embedding objectives, can be reused across diverse 3D segmentation tasks, and it offers an honest ablation of each component. Strengths of the paper include the fair in-house baseline trained with the same data and architecture, the explicit statement that loss weights were fixed rather than tuned on downstream performance, the inclusion of neutral or negative ablation results, and the promise of public model weights. However, the headline improvement is small relative to the reported uncertainty, and the evaluation uses only a single pre-training run per configuration, so the reproducibility of the main claim is not yet demonstrated.
major comments (3)
- [§4.3, Tables 5 and 6] The central claim that Curia-MAE improves frozen-encoder performance over the MAE baseline is not statistically supported. The average gains are 66.8 to 67.5 Dice on anatomy and 63.0 to 64.7 on lesions, yet the reported bootstrap intervals are around ±1.2 to ±3.2 Dice per entry, and Section 4.3 states that 'it does not capture fine-tuning seed variance, as each configuration corresponds to a single pre-training run.' The anatomy gain is smaller than the intervals in either table, and the lesion gain is comparable to the combined intervals. The dataset-level pattern is also mixed: Curia-MAE improves on AMOS-CT and TS-MRI but declines on AMOS-MRI and TS-CT. Because each configuration is a single pre-training run and apparently a single fine-tuning run per task, run-to-run seed variance could plausibly account for the observed differences. The paper's own caveat that 'differences smaller than these intervals should be regarded as indicative rather than conclusive' applies directly to the headline improvement. To establish the abstract's claim, the authors should provide multiple pre-training/fine-tuning seeds with paired significance testing, or otherwise demonstrate that the improvement is reproducible.
- [§4.4, Tables 3 and 4] The component ablation that motivates the method has the same single-run limitation. For example, the Barlow Twins local–global objective is reported as the key frozen-encoder improvement (66.3 vs. 65.1 for the no-objective configuration on anatomy), but the entries carry bootstrap intervals of ±1.2 to ±2.0, so the contribution of the central component is not separated from seed noise. The BYOL comparison is also based on one run; the claim that BYOL 'admits partial representational collapse' is plausible but unsupported without multiple seeds or diagnostic evidence. A paired multi-seed comparison of the final method versus the baseline would also clarify whether the improvement is robust.
- [§4.5, Table 6] The claim of superiority on lesion tasks under full fine-tuning is based on an average of 69.4 for Curia-MAE versus 68.6 for nnU-Net and 69.1 for ResEnc-L, with per-dataset intervals of roughly ±2 to ±5 Dice. This difference is small relative to the reported evaluation uncertainty and is not accompanied by any significance test or seed variance estimate. The claim should be softened or supported with additional evidence, especially because the paper elsewhere emphasizes that small differences should be treated as indicative only.
minor comments (5)
- [§4.3] The sentences defining the bootstrap interval and its limitation are clear, but the paper would benefit from explicitly stating how many fine-tuning runs were performed for each configuration; the current text implies one, yet this is never stated directly.
- [§4.4] The random spacing component is included in the ablation but is described as targeting robustness rather than benchmark accuracy, and no experiment measures resolution robustness directly. If this property is a stated benefit, a simple evaluation under spacing shift would make the claim testable.
- [Eq. (7)] The similarity criterion D is left abstract in the main text; the paper refers to BYOL and Barlow Twins formulations without giving the exact objective used for the final model. A concise equation for the Barlow Twins loss with the chosen projection sizes would improve reproducibility.
- [Table 7] The table reports a large gain on task O4 (49.8 to 63.9 c-index) without any uncertainty interval; given the paper's own warning about small differences, adding bootstrap intervals or at least a paired test would help readers judge whether this improvement is meaningful.
- [Throughout] The phrase 'superior on lesion tasks' in the abstract is stronger than what the full fine-tuning results show; in the conclusion the authors appropriately qualify the results as 'suggest,' and the abstract should be aligned with that qualified framing.
Circularity Check
No material circularity: the central result is a controlled comparison against an in-house MAE baseline trained on the same data and architecture.
full rationale
The paper's central claim is that Curia-MAE improves frozen-encoder segmentation over a strong MAE baseline. That comparison (Tables 5 and 6) is between two models trained by the authors on the same pre-training data and evaluated with the same downstream protocol; the baseline is explicitly retrained under identical full-scale conditions, so the improvement is not an imported or self-referential quantity. The loss components are introduced with fixed weights (β=0.3, λctx=0.2, λreg=0.1, λsim=1.0) and the paper states they were not tuned on downstream benchmarks. Self-citations to Curia [9] and Curia-2 [20] occur in related work and in the CuriaBench3D comparison, but the paper explicitly identifies the baseline comparison as the relevant one and reports other foundation-model scores only for context, so those citations are not load-bearing. The empirical choice of Barlow Twins over BYOL is made on the same downstream benchmarks in Section 4.4, which raises a selection-on-evaluation-data concern rather than a circularity-by-construction issue: the selected objective is not defined in terms of the reported Dice scores, and the final tables still compare against a fixed baseline. Statistical fragility of the headline gains (bootstrap intervals, single pre-training run) is a robustness limitation, not a circular derivation. No equation is defined in terms of the result it is supposed to predict, and no fitted parameter is renamed as a prediction.
Assumptions & free parameters
free parameters (5)
- Huber loss threshold beta =
0.3
- Context loss weight lambda_ctx =
0.2
- SIGReg regularizer weight lambda_reg =
0.1
- Local-global similarity weight lambda_sim =
1.0
- Masking ratio =
0.75
assumptions (3)
- domain assumption The private pre-training dataset of 300,000 CT/MRI volumes is representative of the downstream evaluation distribution and does not overlap with the public benchmarks.
- standard math Pre-training objectives from prior work (MAE, SIGReg, Barlow Twins) behave as described and their known properties hold in the 3D medical setting.
- domain assumption The global view produced by resizing the full volume to the encoder input size preserves enough anatomical consistency for the local-global alignment to be meaningful.
Cite this review
Pith. "Pith review of Curia-MAE: Multi-Modal Multi-Anatomy MAE Pre-Training for 3D Medical Image Segmentation." pith.science (2026). https://pith.science/paper/ORHJJHQL
@misc{pith2026260805844,
author = {Pith},
title = {Pith review of: Curia-MAE: Multi-Modal Multi-Anatomy MAE Pre-Training for 3D Medical Image Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/ORHJJHQL}},
note = {Machine review of arXiv:2608.05844}
}
read the original abstract
Radiology foundation models learn transferable representations that can be adapted to new tasks by training only small layers on top of a frozen encoder. Dense prediction tasks such as 3D segmentation are, however, underrepresented in their evaluation, and, with the encoder kept frozen, pre-trained models still fall short of nnU-Net, the state-of-the-art reference trained from scratch. To close this gap we extend convolutional MAE pre-training with a robust reconstruction objective, a feature regularizer, and a local-global similarity objective. Using this method, we propose Curia-MAE, a multi-modal, multi-anatomy MAE model pre-trained on 300,000 CT and MRI images covering a large number of anatomical sites. On eight anatomy- and lesion-focused segmentation benchmarks, Curia-MAE improves frozen-encoder performance over a strong MAE baseline, while remaining competitive under full finetuning and superior on lesion tasks, where labeled data is scarce. These results indicate that a single frozen encoder can be reused across diverse segmentation tasks, reducing the cost of adapting and deploying such models in clinical workflows. We will make our pre-trained model weights publicly available.
Figures
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:2511.17803 (2025) 1, 13
Agrawal, K.K., Liu, L., Lian, L., Nercessian, M., Harguindeguy, N., Wu, Y., Mikhael, P., Lin, G., Sequist, L.V., Fintelmann, F., et al.: Pillar-0: A new frontier for radiology foundation models. arXiv preprint arXiv:2511.17803 (2025) 1, 13
arXiv 2025
-
[2]
Akinci D’Antonoli, T., Berger, L.K., Indrakanti, A.K., Vishwanathan, N., Weiss, J., Jung, M., Berkarda, Z., Rau, A., Reisert, M., Küstner, T., Walter, A., Merkle, E.M., Boll, D.T., Breit, H.C., Nicoli, A.P., Segeroth, M., Cyriac, J., Yang, S., Wasserthal, J.: Totalsegmentator mri: Robust sequence-independent segmentation of multiple anatomic structures in...
-
[3]
Nature communications13(1), 4128 (2022) 8, 14
Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B.A., Litjens, G., Menze, B., Ronneberger, O., Summers, R.M., et al.: The medical segmentation decathlon. Nature communications13(1), 4128 (2022) 8, 14
work page 2022
-
[4]
arXiv preprint arXiv:2511.08544 (2025) 5, 6
Balestriero, R., LeCun, Y.: Lejepa: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544 (2025) 5, 6
arXiv 2025
-
[5]
Blankemeier, L., Cohen, J.P., Kumar, A., Van Veen, D., Gardezi, S.J.S., Paschali, M., Chen, Z., Delbrouck, J.B., Reis, E., Truyts, C., et al.: Merlin: A vision language foundation model for 3d computed tomography. Research Square pp. rs–3 (2024) 1, 13
work page 2024
-
[6]
In: Proceedings of the IEEE/CVF international conference on computer vision
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9650–9660 (2021) 2
work page 2021
-
[7]
In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision
Chen, Z., Agarwal, D., Aggarwal, K., Safta, W., Balan, M.M., Brown, K.: Masked image modeling advances 3d medical image analysis. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1970– 1980 (2023) 3
work page 2023
-
[8]
Codella, N.C.F., Jin, Y., Jain, S., Gu, Y., Lee, H.H., Ben Abacha, A., Santamaria- Pang, A., Guyman, W., Sangani, N., Zhang, S., Poon, H., Hyland, S., Bannur, S., Alvarez-Valle, J., Li, X., Garrett, J., McMillan, A., Rajguru, G., Maddi, M., Vijayrania, N., Bhimai, R., Mecklenburg, N., Jain, R., Holstein, D., Gaur, N., Aski, V., Hwang, J.N., Lin, T., Tarap...
work page 2024
Show all 30 references
-
[9]
arXiv preprint arXiv:2509.06830 (2025) 1, 2, 7, 13 16 T
Dancette, C., Khlaut, J., Saporta, A., Philippe, H., Ferreres, E., Callard, B., Danielou, T., Alberge, L., Machado, L., Tordjman, D., et al.: Curia: A multi- modal foundation model for radiology. arXiv preprint arXiv:2509.06830 (2025) 1, 2, 7, 13 16 T. Danielou et al
2025 arXiv
-
[10]
Advances in neural information processing systems33, 21271–21284 (2020) 2
Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Do- ersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al.: Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems33, 2127...
2020
-
[11]
In: Advances in Neural Information Processing Systems (NeurIPS) (2020) 7
Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P.H., Buchatskaya, E., Doersch, C., Pires, B.A., Guo, Z.D., Azar, M.G., et al.: Bootstrap your own latent: A new approach to self-supervised learning. In: Advances in Neural Information Processing Systems (NeurIPS) (2020) 7
2020
-
[12]
arXiv preprint arXiv:2403.17834 (2024) 13
Hamamci, I.E., Er, S., Menze, B.: Ct-clip, a pre-trained text-image model for 3d chest ct image and radiology report analysis. arXiv preprint arXiv:2403.17834 (2024) 13
2024
-
[13]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16000–16009 (2022) 3
2022
-
[14]
Nature methods18(2), 203–211 (2021) 2, 3, 9, 11, 12, 14
Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods18(2), 203–211 (2021) 2, 3, 9, 11, 12, 14
2021
-
[15]
In: International Conference on Medical Image Computing and Computer- Assisted Intervention
Isensee, F., Wald, T., Ulrich, C., Baumgartner, M., Roy, S., Maier-Hein, K., Jaeger, P.F.: nnu-net revisited: A call for rigorous validation in 3d medical image segmen- tation. In: International Conference on Medical Image Computing and Computer- Assisted Intervention. pp. 488...
2024
-
[16]
arXiv preprint arXiv:2206.08023 (2022) 8
Ji, Y., Bai, H., Yang, J., Ge, C., Zhu, Y., Zhang, R., Li, Z., Zhang, L., Ma, W., Wan, X., et al.: Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. arXiv preprint arXiv:2206.08023 (2022) 8
2022 arXiv
-
[17]
Scientific data9(1), 320 (2022) 8
Liew, S.L., Lo, B.P., Donnelly, M.R., Zavaliangos-Petropulu, A., Jeong, J.N., Barisano, G., Hutton, A., Simon, J.P., Juliano, J.M., Suri, A., et al.: A large, curated, open-source stroke neuroimaging dataset to improve lesion segmentation algorithms. Scientific data9(1), 320 (2022) 8
2022
-
[18]
Munk, A., Ambsdorf, J., Llambias, S., Nielsen, M.: Amaes: Augmented masked autoencoderpretrainingonpublicbrainmridatafor3d-nativesegmentation.arXiv preprint arXiv:2408.00640 (2024) 2, 3
2024 arXiv
-
[19]
In: International conference on machine learning
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021) 2
2021
-
[20]
arXiv preprint arXiv:2604.01987 (2026) 1, 2, 7, 13
Saporta,A.,Callard,B.,Dancette,C.,Khlaut,J.,Corbière,C.,Butsanets,L.,Prat, A., Manceron, P.: Curia-2: Scaling self-supervised learning for radiology foundation models. arXiv preprint arXiv:2604.01987 (2026) 1, 2, 7, 13
2026
-
[21]
arXiv preprint arXiv:2507.05201 (2025) 1, 13
Sellergren, A., Kazemzadeh, S., Jaroensri, T., Kiraly, A., Traverse, M., Kohlberger, T., Xu, S., Jamil, F., Hughes, C., Lau, C., et al.: Medgemma technical report. arXiv preprint arXiv:2507.05201 (2025) 1, 13
2025 arXiv
-
[22]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Tang, Y., Yang, D., Li, W., Roth, H.R., Landman, B., Xu, D., Nath, V., Hatamizadeh, A.: Self-supervised pre-training of swin transformers for 3d med- ical image analysis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 20730–20740 (2022) 3
2022
-
[23]
arXiv preprint arXiv:2301.03580 (2023) 3
Tian, K., Jiang, Y., Diao, Q., Lin, C., Wang, L., Yuan, Z.: Designing bert for convolutional networks: Sparse and hierarchical masked modeling. arXiv preprint arXiv:2301.03580 (2023) 3
2023 arXiv
-
[24]
In: Proceedings of the Computer Vision and Pattern Recognition Conference
Wald, T., Ulrich, C., Lukyanenko, S., Goncharov, A., Paderno, A., Miller, M., Maerkisch, L., Jaeger, P., Maier-Hein, K.: Revisiting mae pre-training for 3d med- Multi-Modal Multi-Anatomy MAE Training for Medical Segmentation 17 ical image segmentation. In: Proceedings of the C...
2025
-
[25]
Wasserthal, J.: Dataset with segmentations of 117 important anatomical structures in 1228 ct images (2023).https://doi.org/10.5281/zenodo.10047292,https: //doi.org/10.5281/zenodo.100472928
2023 doi
-
[26]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Woo, S., Debnath, S., Hu, R., Chen, X., Liu, Z., Kweon, I.S., Xie, S.: Convnext v2: Co-designing and scaling convnets with masked autoencoders. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16133– 16142 (2023) 3
2023
-
[27]
In: International conference on machine learn- ing
Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow twins: Self-supervised learning via redundancy reduction. In: International conference on machine learn- ing. pp. 12310–12320. PMLR (2021) 2
2021
-
[28]
In: International Conference on Machine Learn- ing (ICML) (2021) 7
Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow twins: Self-supervised learning via redundancy reduction. In: International Conference on Machine Learn- ing (ICML) (2021) 7
2021
-
[29]
NEJM AI2(1) (Jan 2025) 1, 2, 13
Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., Wong, C., Tupini, A., Wang, Y., Mazzola, M., Shukla, S., Liden, L., Gao, J., Crabtree, A., Piening, B., Bifulco, C., Lungren, M.P., Naumann, T., Wang, S., Poon, H.: Biomedc...
2025
-
[30]
IEEE Transac- tions on Medical Imaging (2025) 3
Zhuang, J., Luo, L., Wang, Q., Wu, M., Luo, L., Chen, H.: Advancing volumetric medical image segmentation via global-local masked autoencoders. IEEE Transac- tions on Medical Imaging (2025) 3
2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.