REVIEW 4 major objections 5 minor 23 references
Benchmarking GANs, Diffusion Models, and Flow Matching for T1w-to-T2w MRI Translation
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that Pix2Pix, a conditional GAN, outperforms diffusion and flow matching models for T1w-to-T2w MRI translation across three public datasets when all models share the same U-Net backbone and training budget.
desk verdict Useful reproducible benchmark, but Table 1's MSE and PSNR are mathematically inconsistent, so the central ranking is not credible until recomputed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The comparison is carried by a deliberate equalization: every model sits on the same 2D U-Net encoder-decoder backbone with skip connections, two residual blocks per resolution level, channel widths of 32, 64, 64, and 64, and attention at the last two resolution levels, and every model gets the same subject-level 80/5/15 split, batch size of 6, and up to 300 epochs with early stopping on validation loss. What differs between families is the training loss and conditioning scheme: Pix2Pix minimizes an adversarial loss plus a weighted L1 reconstruction loss; the diffusion model minimizes noise-prediction error with the T1w slice concatenated or injected through a ControlNet-style branch; the flow matching model regresses a conditional velocity field along a straight line from noise or directly from the T1w slice. This shared architecture and training budget is what turns the results into an inter-family ranking rather than a comparison of unrelated implementations.
What would settle it
Re-run the benchmark on the same three datasets but without early stopping, extending training to more epochs, widening the backbone's channel counts, and tuning each family's noise schedule and step count; if flow matching or diffusion then reaches or exceeds Pix2Pix's SSIM of 0.862 and PSNR of 22.915, the reported superiority is an artifact of the shared budget. A faster check is to plot validation losses across training: if the FM and diffusion curves are still descending when early stopping fires, the comparison stopped those families before they converged.
Extended reading notes
Core claim
The paper's central discovery, on its own terms, is that Pix2Pix is not merely acceptable but dominant in this regime. With a shared 2D U-Net backbone and a 300-epoch, early-stopped budget, Pix2Pix reaches SSIM 0.862, PSNR 22.915 dB, and MSE 0.0054, while the best flow-matching variant reaches SSIM 0.732 and the best diffusion variant reaches SSIM 0.469. Pix2Pix inference takes 0.05 seconds; flow matching variants take about 11.6 to 11.9 seconds; diffusion variants take 41 to 58 seconds. Visually, Pix2Pix preserves sharp anatomical boundaries and transfers white-matter lesions, flow matching sometimes inpaints lesions rather than reconstructing them, and diffusion methods blur or fail to transfer lesions. The paper attributes the gap to flow-based models overfitting on small datasets and simpler tasks, and it explicitly cautions that the finding may not generalize to larger datasets, higher-dimensional data, or more complex tasks.
Load-bearing premise
The load-bearing premise is that a single 300-epoch, early-stopped training budget and one shared U-Net backbone with 32-64-64-64 channels are representative configurations for all three model families; if diffusion or flow matching models need more training, more capacity, or adjusted schedules to reach their typical performance, the reported ranking describes the recipe, not an intrinsic winner.
Editorial extensions
If this is right
- In low-data, 2D-slice MRI translation, clinical deployment should favor a GAN with a reconstruction loss: Pix2Pix gives the best fidelity and a per-slice inference cost of 0.05 seconds, suitable for near-real-time use.
- Flow matching is the middle ground: Direct FM's higher SSIM (0.732 versus 0.715) suggests the T1w slice is more useful as the flow source than as a concatenated condition, but both FM variants remain roughly two orders of magnitude slower than Pix2Pix.
- The diffusion baselines, as configured with 1000 noise steps and a linear schedule, are not clinically usable here, with SSIM values of 0.469 and 0.363 indicating substantial anatomical distortion.
- The overfitting explanation implies that data volume is the key lever: flow and diffusion methods may close the gap as datasets grow, so the reported ranking is a small-data result rather than a universal ordering.
- The authors state their conclusion is limited to this setting and may not carry over to larger datasets, higher-dimensional 3D volumes, or more complex translation tasks.
Reading between the lines
- An unstated consequence is that this benchmark should not be read as GANs being better than diffusion on MRI in general; it is evidence about one fixed recipe, and that recipe may underpower the iterative methods by stopping them before convergence.
- A direct test of the overfitting story is to train each family on progressively larger subsets of the same data, for example 25, 50, and 100 percent, and check whether the FM-to-Pix2Pix gap shrinks monotonically; the paper's explanation predicts that it will.
- Neighboring problems that would stress the ranking are 3D volumetric translation, translation between contrasts with weaker structural correlation, and evaluation by downstream tasks such as lesion segmentation or radiologist preference, which raw SSIM, MSE, and PSNR do not measure.
- Because the paper reports only aggregate means and standard deviations, a per-subject or per-slice breakdown could reveal whether Pix2Pix's margin is consistent or dominated by a few hard cases.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports a benchmark of three generative model families—GANs (Pix2Pix), conditional diffusion models (with input concatenation and with a ControlNet module), and flow matching (with concatenated noise and with direct T1w initialization)—for 2D axial T1w-to-T2w MRI translation. Experiments use three public datasets (IXI, HCP, CamCAN), a shared U-Net backbone, and a common training budget of up to 300 epochs with early stopping. The authors report SSIM, MSE, and PSNR, show qualitative examples for two test subjects, and compare inference time, parameters, and memory. The central claim is that Pix2Pix outperforms diffusion- and FM-based methods on structural fidelity, image quality, and computational efficiency, with an additional interpretive claim that flow-based models overfit on small datasets.
Significance. If the quantitative results are correct, this is a useful practical benchmark for an active clinical imaging task: it compares three model families under a unified implementation, uses publicly available datasets, makes code and models available, and reports resource usage in addition to quality metrics. The contribution is empirical rather than methodological, so its value depends heavily on the reliability of Table 1 and on whether the 'comparable settings' claim is actually satisfied. The overfitting interpretation in the abstract and conclusion is not supported by the reported measurements and should be reframed as a hypothesis.
major comments (4)
- [Table 1 / Section 4.3] The reported MSE and PSNR values are internally inconsistent under the stated aggregation. Section 4.3 says SSIM, MSE, and PSNR are used to assess performance, and the Table 1 caption says metrics are 'mean ± standard deviation across the test set.' For images normalized to [0,1], PSNR_i = -10*log10(MSE_i). Under this standard relation, Pix2Pix's MSE = 0.0054 ± 0.0000 cannot co-exist with PSNR = 22.915 ± 3.428: a zero MSE standard deviation at four decimals constrains the per-subject MSE to roughly [0.00535, 0.00545], which yields a PSNR spread of about 0.08 dB, not 3.428 dB. The same contradiction appears for Concat. FM (MSE 0.0097 ± 0.0000 with PSNR std 1.143) and Direct FM (MSE 0.0106 ± 0.0000 with PSNR std 1.361). The mean values are also inconsistent for the diffusion rows: under [0,1] normalization, MSE = 0.5110 corresponds to PSNR ≈ 2.9 dB, not 15.892 dB, and MSE = 0.3887 corresponds to ≈ 4.1 dB, not 14.800 dB. Either the metrics are computed on different scales or with different aggregations (e.g., PSNR averaged per image while MSE is pooled over the whole test set), contradicting the stated protocol, or the numbers are erroneous. Because Table 1 is the principal evidence for the central claim, this must be corrected before the ranking can be assessed.
- [Section 3.2 / Section 4.3] The ControlNet configuration violates the paper's 'comparable settings' claim. Section 3.2 states that ControlNet is trained on top of a 'pre-trained, frozen unconditional diffusion backbone,' while Section 4.3 says that 'each model employs a 2D U-Net encoder-decoder backbone' and Section 4.4 says that 'we train all networks' for up to 300 epochs with early stopping. A frozen pre-trained backbone is not trained under the same protocol as the other models, and Table 2 shows ControlNet has 3,255,745 parameters versus 2,328,737 for the other diffusion and FM models. The authors should either train all diffusion variants from scratch under identical conditions or explicitly report the ControlNet backbone's training setup and restrict the comparability claim to the models that were actually trained under the shared protocol.
- [Abstract / Section 5.1 / Section 6] The claim that flow-based models 'are prone to overfitting on small datasets and simpler tasks' is not supported by the measurements in this paper. Table 1 reports only test-set SSIM, MSE, and PSNR; there is no training-versus-test gap, no memorization metric, and no dataset-size ablation. Lower test performance alone does not establish overfitting, and references [4,1] are external evidence rather than evidence from this study. The authors should either provide direct evidence of overfitting (for example, train/test gap curves, sample-memorization metrics, or an ablation across dataset sizes) or soften the wording to state that this is a hypothesis consistent with prior work.
- [Section 5.1] The quantitative comparison is presented only as aggregate means over three datasets. Because the three datasets differ in acquisition protocol, size, and subject population, and because the paper claims a 'comprehensive benchmark,' the authors should report per-dataset results and, where appropriate, paired significance tests or confidence intervals for the Pix2Pix-versus-FM and Pix2Pix-versus-diffusion comparisons. Without this, the word 'demonstrably' in the description of Pix2Pix's superiority is not statistically supported.
minor comments (5)
- [Section 4.2 / Table 1] The normalization range used for PSNR is never stated. Please specify whether pixel values are in [0,1] or [0,255] and provide the explicit PSNR formula used, so that Table 1 can be interpreted unambiguously.
- [Section 5.2] The qualitative claims, such as 'most clinically acceptable and realistic synthetic images,' are based on visual inspection of two test subjects. No reader study, blinded evaluation, or inter-rater assessment is described. I suggest tempering the language or adding a small reader study to support claims of clinical acceptability.
- [Section 5.3 / Table 2] The inference-time comparison conflates the chosen number of sampling steps with the model family: diffusion uses 1000 steps and FM uses 300 Euler steps, so the 0.05 s versus 11–12 s comparison is partly a step-count artifact. Please report time per function evaluation or add a discussion of how the ranking changes with matched step counts.
- [Various] There are several typographical and formatting issues, including 'T able 1' in the caption, missing spacing in 'Lcf m' in Section 3.3, and inconsistent use of 'FM' versus 'flow matching' in the conclusions. These do not affect the technical content but should be cleaned up.
- [Abstract / Conclusion] The conclusion correctly notes that the findings 'may not generalize to other settings,' but the abstract states the ranking without this scope qualifier. Aligning the abstract with the stated limitations would improve accuracy.
Circularity Check
No circularity: the benchmark ranking is an empirical comparison against held-out test data, not a derivation from fitted inputs.
full rationale
The paper's central claim that Pix2Pix outperforms diffusion and flow-matching methods on T1w-to-T2w translation is supported by quantitative metrics (SSIM, MSE, PSNR) computed on a held-out test split with strict subject-level separation, as described in Section 4.3. The compared model families use standard, externally published training objectives (Pix2Pix [13], DDPM [10], Flow Matching [15]) and public datasets (IXI, HCP, CamCAN); no parameter is fitted to the test set and then renamed a prediction. The only interpretive statements citing prior work (overfitting of flow models on small datasets, refs [1,4,9]) are external support for the observed pattern, not inputs that force the reported ranking. The known internal inconsistency in Table 1 between MSE and PSNR values is a data-quality or aggregation issue, not a circularity: it does not make the conclusion equivalent to its inputs by construction. No self-citation appears in the reference list, and no uniqueness theorem or ansatz is imported from the authors' prior work. Therefore the circularity burden is essentially zero.
Assumptions & free parameters
free parameters (5)
- Diffusion forward steps (T) =
1000
- Diffusion noise schedule endpoints (beta) =
1e-4 to 2e-2
- Flow matching Euler solver steps =
300
- Pix2Pix L1 weight (lambda) =
100
- Training budget (epochs) =
up to 300 with early stopping
assumptions (4)
- domain assumption The central axial slice is a sufficient proxy for whole-volume T1w-to-T2w translation quality.
- domain assumption SSIM, MSE, and PSNR capture image fidelity, structural preservation, and quality for this task.
- domain assumption A shared U-Net backbone with the same channel counts is a fair capacity match across GAN, diffusion, and flow matching families.
- domain assumption Early stopping on validation loss selects models fairly for all families.
Cite this review
Pith. "Pith review of Benchmarking GANs, Diffusion Models, and Flow Matching for T1w-to-T2w MRI Translation." pith.science (2026). https://pith.science/paper/RUZU3IZ5
@misc{pith2026250714575,
author = {Pith},
title = {Pith review of: Benchmarking GANs, Diffusion Models, and Flow Matching for T1w-to-T2w MRI Translation},
year = {2026},
howpublished = {\url{https://pith.science/paper/RUZU3IZ5}},
note = {Machine review of arXiv:2507.14575}
}
abstract
Magnetic Resonance Imaging (MRI) enables the acquisition of multiple image contrasts, such as T1-weighted (T1w) and T2-weighted (T2w) scans, each offering distinct diagnostic insights. However, acquiring all desired modalities increases scan time and cost, motivating research into computational methods for cross-modal synthesis. To address this, recent approaches aim to synthesize missing MRI contrasts from those already acquired, reducing acquisition time while preserving diagnostic quality. Image-to-image (I2I) translation provides a promising framework for this task. In this paper, we present a comprehensive benchmark of generative models$\unicode{x2013}$specifically, Generative Adversarial Networks (GANs), diffusion models, and flow matching (FM) techniques$\unicode{x2013}$for T1w-to-T2w 2D MRI I2I translation. All frameworks are implemented with comparable settings and evaluated on three publicly available MRI datasets of healthy adults. Our quantitative and qualitative analyses show that the GAN-based Pix2Pix model outperforms diffusion and FM-based methods in terms of structural fidelity, image quality, and computational efficiency. Consistent with existing literature, these results suggest that flow-based models are prone to overfitting on small datasets and simpler tasks, and may require more data to match or surpass GAN performance. These findings offer practical guidance for deploying I2I translation techniques in real-world MRI workflows and highlight promising directions for future research in cross-modal medical image synthesis. Code and models are publicly available at https://github.com/AndreaMoschetto/medical-I2I-benchmark.
Figures
Reference graph
Works this paper leans on
-
[1]
Machine Learning: Science and Technology6(1), 015022 (2025)
Akbar, M.U., Wang, W., Eklund, A.: Beware of diffusion models for synthesizing medical images—a comparison with gans in terms of memorizing brain mri and chest x-ray images. Machine Learning: Science and Technology6(1), 015022 (2025)
work page 2025
-
[2]
Alkan, C., Cocjin, J., Weitz, A.: Magnetic resonance contrast prediction using deep learning. Google Scholar (2016)
work page 2016
-
[3]
Medical image analysis 12(1), 26–41 (2008)
Avants, B.B., Epstein, C.L., Grossman, M., Gee, J.C.: Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Medical image analysis 12(1), 26–41 (2008)
work page 2008
-
[4]
arXiv preprint arXiv:2506.03719 (2025)
Bertrand, Q., Gagneux, A., Massias, M., Emonet, R.: On the closed-form of flow matching: Generalization does not arise from target stochasticity. arXiv preprint arXiv:2506.03719 (2025)
arXiv 2025
-
[5]
arXiv preprint arXiv:2211.02701 (2022)
Cardoso, M.J., Li, W., Brown, R., Ma, N., Kerfoot, E., Wang, Y., Murrey, B., Myronenko, A., Zhao, C., Yang, D., et al.: Monai: An open-source framework for deep learning in healthcare. arXiv preprint arXiv:2211.02701 (2022)
arXiv 2022
-
[6]
arXiv preprint arXiv:2503.07535 (2025) 12 A
Chadebec, C., Tasar, O., Sreetharan, S., Aubin, B.: Lbm: Latent bridge matching for fast image-to-image translation. arXiv preprint arXiv:2503.07535 (2025) 12 A. Moschetto et al
arXiv 2025
-
[7]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Choo, K., Jun, Y., Yun, M., Hwang, S.J.: Slice-consistent 3d volumetric brain ct-to-mri translation with 2d brownian bridge diffusion model. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 657–667. Springer (2024)
2024
-
[8]
In: Medical Imaging with Deep Learning
Dey, A., Ebrahimi, M.: Mtsr-mri: Combined modality translation and super- resolution of magnetic resonance images. In: Medical Imaging with Deep Learning. pp. 743–757. PMLR (2024)
work page 2024
Show all 23 references
-
[9]
Advances in neural information processing systems 34, 8780–8794 (2021)
Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34, 8780–8794 (2021)
2021
-
[10]
Advances in neural information processing systems 33, 6840–6851 (2020)
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020)
2020
-
[11]
NeuroImage 260, 119474 (2022)
Hoopes, A., Mora, J.S., Dalca, A.V., Fischl, B., Hoffmann, M.: Synthstrip: skull- stripping for any brain image. NeuroImage 260, 119474 (2022)
2022
-
[12]
In: ECCV
Huang, Y., Zheng, F., Sun, X., Li, Y., Shao, L., Zheng, Y.: Generalized brain image synthesis with transferable convolutional sparse coding networks. In: ECCV. pp. 183–199. Springer (2022)
2022
-
[13]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with condi- tional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1125–1134 (2017)
2017
-
[14]
reports of practical Oncology and radiotherapy 26(1), 35–42 (2021)
Kawahara, D., Nagata, Y.: T1-weighted and t2-weighted mri image synthesis with convolutional generative adversarial networks. reports of practical Oncology and radiotherapy 26(1), 35–42 (2021)
2021
-
[15]
arXiv preprint arXiv:2210.02747 (2022)
Lipman, Y., Chen, R.T., Ben-Hamu, H., Nickel, M., Le, M.: Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 (2022)
2022 arXiv
-
[16]
arXiv preprint arXiv:2410.02423 (2024)
Martin, S., Gagneux, A., Hagemann, P., Steidl, G.: Pnp-flow: Plug-and-play image restoration with flow matching. arXiv preprint arXiv:2410.02423 (2024)
2024 arXiv
-
[17]
Medical Physics 51(4), 2538–2548 (2024)
Pan, S., Abouei, E., Wynne, J., Chang, C.W., Wang, T., Qiu, R.L., Li, Y., Peng, J., Roper, J., Patel, P., et al.: Synthetic ct generation from mri using 3d transformer- based denoising diffusion model. Medical Physics 51(4), 2538–2548 (2024)
2024
-
[18]
In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18....
2015
-
[19]
NeuroImage: Clinical 6, 9–19 (2014)
Shinohara, R.T., Sweeney, E.M., Goldsmith, J., Shiee, N., Mateen, F.J., Calabresi, P.A., Jarso, S., Pham, D.L., Reich, D.S., Crainiceanu, C.M., et al.: Statistical normalization techniques for magnetic resonance imaging. NeuroImage: Clinical 6, 9–19 (2014)
2014
-
[20]
IEEE transactions on medical imag- ing 29(6), 1310–1320 (2010)
Tustison, N.J., Avants, B.B., Cook, P.A., Zheng, Y., Egan, A., Yushkevich, P.A., Gee, J.C.: N4itk: improved n3 bias correction. IEEE transactions on medical imag- ing 29(6), 1310–1320 (2010)
2010
-
[21]
In: Medical Imaging 2022: Image Processing
Vaidya, A., Stough, J.V., Patel, A.A.: Perceptually improved t1-t2 mri translations using conditional generative adversarial networks. In: Medical Imaging 2022: Image Processing. vol. 12032, pp. 505–511. SPIE (2022)
2022
-
[22]
arXiv preprint arXiv:2503.00266 (2025)
Yazdani, M., Medghalchi, Y., Ashrafian, P., Hacihaliloglu, I., Shahriari, D.: Flow matching for medical image synthesis: Bridging the gap between speed and quality. arXiv preprint arXiv:2503.00266 (2025)
2025 arXiv
-
[23]
In: Proceedings of the IEEE/CVF international conference on computer vision
Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 3836–3847 (2023)
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.