Improving Combined Detection and Classification of TEM Defects via Mask-Conditioned Latent Diffusion Augmentation
Pith reviewed 2026-06-28 14:42 UTC · model grok-4.3
The pith
Augmenting small TEM datasets with mask-conditioned latent diffusion generated images improves combined defect detection and classification F1 by up to 0.02.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Mask-conditioned latent diffusion models can produce realistic synthetic TEM images with automatic multi-class defect masks that, when used to augment small experimental datasets, yield up to a 0.02 improvement in the harmonic mean of detection and classification F1 scores for a Mask R-CNN model.
What carries the argument
Mask-conditioned latent diffusion model that learns distributions from experimental masks to generate controllable synthetic image-mask pairs for data augmentation.
If this is right
- Generative augmentation provides small performance boosts on datasets as small as 10 labeled images.
- The improvement in detection versus classification tasks varies with the specific train and test data split.
- Generation requires no additional manual annotations beyond the initial experimental masks.
- Targeted generative models can support deep learning in data-scarce microscopy quantification.
Where Pith is reading between the lines
- This method could be tested on other types of microscopy images if mask patterns are similar.
- Further work might check whether the gains hold when base datasets are larger than 100 images.
- The approach reduces reliance on expert time for labeling new defect instances.
Load-bearing premise
The synthetic images match real TEM images closely enough that adding them improves performance on real test data rather than causing the model to learn synthetic-specific features.
What would settle it
If a model trained on the augmented data shows lower or equal harmonic mean F1 on real held-out test images compared to training on real data alone, the claimed improvement would not hold.
Figures
read the original abstract
Analyzing microstructural defects in transmission electron microscopy (TEM) images, particularly in irradiated metal alloys, is often limited by the availability of high-quality, labeled data. To address this, we introduce a generative data augmentation approach using a mask-conditioned latent diffusion model (LDM) for synthesizing realistic TEM images with controllable, automatically labeled multi-class defect masks. Without requiring manual annotations for generation, our method enables the creation of synthetic image-mask pairs by sampling distributions learned from experimental masks. These generated data were used to augment small experimental datasets of varying sizes (10, 50, and 100 labeled experimental images) to train a Mask Regional Convolutional Neural Network (R-CNN) model for defect detection and classification. Our results show that generative augmentation yields small overall model performance improvements, with up to a 0.02 gain in the harmonic mean of detection and classification F1 scores. However, we also find that the relative contributions to detection and classification improvement depend on the specific train/test data split. These findings highlight the potential of targeted generative models to enhance deep learning performance in data-scarce microscopy-based image quantification tasks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a mask-conditioned latent diffusion model (LDM) to synthesize realistic TEM images paired with multi-class defect masks from learned distributions of experimental masks. These synthetic pairs augment small real training sets (sizes 10/50/100) to improve Mask R-CNN performance on combined defect detection and classification. Results report modest gains, with a maximum 0.02 increase in the harmonic mean of detection and classification F1 scores, though relative contributions vary by train/test split.
Significance. If the reported gains prove robust and attributable to the augmentation rather than artifacts, the method offers a practical way to address data scarcity in TEM-based materials analysis without manual annotation for synthetics. The mask-conditioning approach enables controllable generation, which is well-suited to the domain. Concrete numeric results on experimental data are a strength, but the small magnitude and split dependence limit broader significance unless further controls are added.
major comments (2)
- [Abstract] Abstract (results paragraph): the claim that generative augmentation produces a genuine 0.02 harmonic-mean F1 lift is load-bearing for the central contribution, yet no quantitative distribution-similarity metric (FID, LPIPS, or defect-statistic match) between real and LDM-generated images is provided; without it the modest gain could arise from split-specific overlap rather than improved training data.
- [Results] Results (split-dependence discussion): the observation that relative detection vs. classification gains depend on the specific train/test split directly undermines robustness of the augmentation benefit; an ablation using deliberately mismatched generated masks (as suggested by the stress-test note) is required to rule out coincidental correlation with the chosen test distribution.
minor comments (1)
- [Abstract] Abstract: the phrase 'up to a 0.02 gain' should be accompanied by the exact baseline and augmented F1 values per split for immediate clarity.
Simulated Author's Rebuttal
We thank the referee for their constructive comments, which help clarify the robustness of our claims. We address each major comment below and have updated the manuscript accordingly.
read point-by-point responses
-
Referee: [Abstract] Abstract (results paragraph): the claim that generative augmentation produces a genuine 0.02 harmonic-mean F1 lift is load-bearing for the central contribution, yet no quantitative distribution-similarity metric (FID, LPIPS, or defect-statistic match) between real and LDM-generated images is provided; without it the modest gain could arise from split-specific overlap rather than improved training data.
Authors: We agree that the absence of explicit distribution-similarity metrics leaves open the possibility that gains arise from split-specific effects rather than data quality. In the revised manuscript we will add FID scores computed on real versus generated images, along with comparisons of defect size, density, and class distributions, to provide quantitative support for the claim. revision: yes
-
Referee: [Results] Results (split-dependence discussion): the observation that relative detection vs. classification gains depend on the specific train/test split directly undermines robustness of the augmentation benefit; an ablation using deliberately mismatched generated masks (as suggested by the stress-test note) is required to rule out coincidental correlation with the chosen test distribution.
Authors: The split dependence is already noted in the manuscript, but we concur that it weakens the robustness argument without further controls. We will therefore perform the requested ablation with deliberately mismatched generated masks and report the outcomes to demonstrate that performance improvements are not explained by coincidental alignment with the test distribution. revision: yes
Circularity Check
No circularity: purely empirical ML augmentation experiment
full rationale
The paper reports measured F1 improvements on held-out real test images after augmenting small real training sets with LDM-generated images. No derivation, equation, or first-principles claim is presented that reduces the reported performance numbers to a fitted parameter, self-citation, or input by construction. The central result is an empirical observation whose validity rests on the experimental protocol (train/test splits, model training) rather than any internal definitional loop. This matches the default case of a self-contained empirical study.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
Was.Fundamentals of Radiation Materials Science: Metals and Alloys
Gary S. Was.Fundamentals of Radiation Materials Science: Metals and Alloys. Springer, New York, NY , 2nd edition, 2016. ISBN 978-1-4939-3437-4. doi: 10.1007/978-1-4939-3438-1. URL https://doi.org/10.1007/ 978-1-4939-3438-1
-
[2]
Faster r-cnn: Towards real-time object detection with region proposal networks, 2016
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks, 2016. arXiv:1506.01497
Pith/arXiv arXiv 2016
-
[3]
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn, 2018. arXiv:1703.06870
Pith/arXiv arXiv 2018
-
[4]
You only look once: Unified, real-time object detection, 2016
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection, 2016. arXiv:1506.02640
Pith/arXiv arXiv 2016
-
[5]
Mingren Shen, Guanzhao Li, Dongxia Wu, Yuhan Liu, Jacob R.C. Greaves, Wei Hao, Nathaniel J. Krakauer, Leah Krudy, Jacob Perez, Varun Sreenivasan, Bryan Sanchez, Oigimer Torres-Velázquez, Wei Li, Kevin G. Field, and Dane Morgan. Multi defect detection and analysis of electron microscopy images with deep learning. Computational Materials Science, 199:110576...
-
[6]
Greaves, Donglin Wang, Zeming Xie, Zitong Huang, Chao Wang, Kevin G
Ryan Jacobs, Mingren Shen, Yuhan Liu, Wei Hao, Xiaoshan Li, Ruoyu He, Jacob R.C. Greaves, Donglin Wang, Zeming Xie, Zitong Huang, Chao Wang, Kevin G. Field, and Dane Morgan. Performance and limitations of deep learning semantic segmentation of multiple defects in transmission electron micrographs.Cell Reports Physical Science, 3(5):100876, 2022. ISSN 2666...
-
[7]
Qinyun Chen, Chaohui Zheng, Yue Cui, Yan-Ru Lin, and Steven J. Zinkle. A deep learning model for automatic analysis of cavities in irradiated materials.Computational Materials Science, 221:112073, 2023. ISSN 0927-0256. doi: https://doi.org/10.1016/j.commatsci.2023.112073. URL https://www.sciencedirect.com/science/ article/pii/S0927025623000678
-
[8]
Mingren Shen, Guanzhao Li, Dongxia Wu, Yudai Yaguchi, Jack C. Haley, Kevin G. Field, and Dane Morgan. A deep learning based automatic defect analysis framework for in-situ tem ion irradiations.Computational Materials Science, 197:110560, 2021. ISSN 0927-0256. doi: https://doi.org/10.1016/j.commatsci.2021.110560. URL https://www.sciencedirect.com/science/a...
-
[9]
Lynch, Steven Chen, Dane Morgan, and Kevin G
Ryan Jacobs, Priyam Patki, Matthew J. Lynch, Steven Chen, Dane Morgan, and Kevin G. Field. Materials swelling revealed through automated semantic segmentation of cavities in electron microscopy images.Scientific Reports, 13(1):5178, Mar 2023. ISSN 2045-2322. doi: 10.1038/s41598-023-32454-2. URL https://doi.org/10. 1038/s41598-023-32454-2
-
[10]
Ryan Jacobs. Deep learning object detection in materials science: Current state and future directions.Computa- tional Materials Science, 211:111527, 2022. ISSN 0927-0256. doi: https://doi.org/10.1016/j.commatsci.2022. 111527. URLhttps://www.sciencedirect.com/science/article/pii/S0927025622002804
-
[11]
Kevin G Field, Priyam Patki, Nasir Sharaf, Kai Sun, Laura Hawkins, Matthew Lynch, Ryan Jacobs, Dane D Morgan, Lingfeng He, and Christopher R Field. Real-time, on-microscope automated quantification of features in microcopy experiments using machine learning and edge computing.Microscopy and Microanalysis, 28(S1): 2046–2048, 08 2022. ISSN 1431-9276. doi: 1...
-
[12]
K. G. Field, R. Jacobs, Shen Mingen, M. Lynch, P. Patki, C. Field, and D. Morgan. Development and deployment of automated machine learning detection in electron microcopy experiments.Microscopy and Microanalysis, 27 (suppl.1):2136–7, 2021. ISSN 1431-9276. doi: 10.1017/S1431927621007704
-
[13]
Wei Li, Kevin G. Field, and Dane Morgan. Automated defect analysis in electron microscopic images.npj Computational Materials, 4(1):36, Jul 2018. ISSN 2057-3960. doi: 10.1038/s41524-018-0093-8. URL https: //doi.org/10.1038/s41524-018-0093-8. 10 Running Title for Header
-
[14]
Holm, Ryan Cohn, Nan Gao, Andrew R
Elizabeth A. Holm, Ryan Cohn, Nan Gao, Andrew R. Kitahara, Thomas P. Matson, Bo Lei, and Srujana Rao Yarasi. Overview: Computer vision and machine learning for microstructural characterization and analysis.Metallurgical and Materials Transactions A, 51(12):5985–5999, Dec 2020. ISSN 1543-1940. doi: 10.1007/s11661-020-06008-4. URLhttps://doi.org/10.1007/s11...
-
[15]
Learning-based defect recognition for quasi-periodic hrstem images.Micron, 146:103069, 2021
Nik Dennler, Antonio Foncubierta-Rodriguez, Titus Neupert, and Marilyne Sousa. Learning-based defect recognition for quasi-periodic hrstem images.Micron, 146:103069, 2021. ISSN 0968-4328. doi: https: //doi.org/10.1016/j.micron.2021.103069. URL https://www.sciencedirect.com/science/article/pii/ S0968432821000603
-
[16]
Ayse Betul Oktay and Anıl Gurses. Automatic detection, localization and segmentation of nano-particles with deep learning in microscopy images.Micron, 120:113–119, 2019. ISSN 0968-4328. doi: https: //doi.org/10.1016/j.micron.2019.02.009. URL https://www.sciencedirect.com/science/article/pii/ S0968432818304013
-
[17]
John Olamofe, Lijun Qian, and Kevin G. Field. Performance evaluation of image super-resolution for cavity detection in irradiated materials.IEEE Access, 13:68052–68065, 2025. doi: 10.1109/ACCESS.2025.3559443
-
[18]
J. Baderot, M. Grould, D. Misra, N. Clément, A. Hallal, S. Martinez, and J. Foucher. Application of deep-learning based techniques for automatic metrology on scanning and transmission electron microscopy images.Journal of Vacuum Science and Technology B, 40(5):054003, 09 2022. ISSN 2166-2746. doi: 10.1116/6.0001988. URL https://doi.org/10.1116/6.0001988
-
[19]
Michael Wu, Jeremy Sharapov, Matthew Anderson, Yu Lu, and Yaqiao Wu. Quantifying dislocation-type defects in post irradiation examination via transfer learning.Scientific Reports, 15(1):15889, May 2025. ISSN 2045-2322. doi: 10.1038/s41598-025-00238-5. URLhttps://doi.org/10.1038/s41598-025-00238-5
-
[20]
A new paradigm in electron microscopy: Automated microstructure analysis utilizing a dynamic segmentation convolutional neutral network.Materials Today Advances, 21:100468,
Stephen Taller, Luke Scime, and Ty Austin. A new paradigm in electron microscopy: Automated microstructure analysis utilizing a dynamic segmentation convolutional neutral network.Materials Today Advances, 21:100468,
-
[21]
doi: https://doi.org/10.1016/j.mtadv.2024.100468
ISSN 2590-0498. doi: https://doi.org/10.1016/j.mtadv.2024.100468. URL https://www.sciencedirect. com/science/article/pii/S2590049824000055
-
[22]
Matthew J. Patrick, Christopher R. Field, Lauren H. L. Grae, Jeffrey M. Rickman, Kevin G. Field, and Katayun Barmak. A comparative analysis of yolov8 and u-net image segmentation approaches for transmission electron micrographs of polycrystalline thin films.APL Machine Learning, 3(3):036105, 07 2025. ISSN 2770-9019. doi: 10.1063/5.0274266. URLhttps://doi....
-
[23]
Gabriella Bruno, Matthew J Lynch, Ryan Jacobs, Dane D Morgan, and Kevin G Field. Evaluation of Human- Bias in Labeling of Ambiguous Features in Electron Microscopy Machine Learning Models.Microscopy and Microanalysis, 29(Supplement_1):1493–1494, 07 2023. ISSN 1431-9276. doi: 10.1093/micmic/ozad067.767. URLhttps://doi.org/10.1093/micmic/ozad067.767
-
[24]
Uberuaga, Cheng Sun, and Ju Li
Dane Morgan, Ghanshyam Pilania, Adrien Couet, Blas P. Uberuaga, Cheng Sun, and Ju Li. Machine learning in nuclear materials research.Current Opinion in Solid State and Materials Science, 26(2):100975, 2022. ISSN 1359-0286. doi: https://doi.org/10.1016/j.cossms.2021.100975. URL https://www.sciencedirect.com/ science/article/pii/S1359028621000784
-
[25]
Santos, Yuting Luo, Sarbajit Banerjee, and Bai-Xiang Xu
Binbin Lin, Nima Emami, David A. Santos, Yuting Luo, Sarbajit Banerjee, and Bai-Xiang Xu. A deep learned nanowire segmentation model using synthetic data augmentation.npj Computational Materials, 8 (1):88, Apr 2022. ISSN 2057-3960. doi: 10.1038/s41524-022-00767-x. URL https://doi.org/10.1038/ s41524-022-00767-x
-
[26]
M. J. Lynch, R. Jacobs, G. A. Bruno, P. Patki, D. Morgan, and K. G. Field. Accelerating domain-aware electron microscopy analysis using deep learning models with synthetic data and image-wide confidence scoring.npj Computational Materials, 11(1):261, Aug 2025. ISSN 2057-3960. doi: 10.1038/s41524-025-01756-6. URL https://doi.org/10.1038/s41524-025-01756-6
-
[27]
Antón Cid-Mejías, Raúl Alonso-Calvo, Helena Gavilán, José Crespo, and Víctor Maojo. A deep learning approach using synthetic images for segmenting and estimating 3d orientation of nanoparticles in em images.Computer Methods and Programs in Biomedicine, 202:105958, 2021. ISSN 0169-2607. doi: https://doi.org/10.1016/j.cmpb. 2021.105958. URLhttps://www.scien...
-
[28]
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization, 2018. URLhttps://arxiv.org/abs/1710.09412. arXiv:1710.09412
Pith/arXiv arXiv 2018
-
[29]
Generative adversarial networks.Commun
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks.Commun. ACM, 63(11):139–144, October 2020. ISSN 0001-0782. doi: 10.1145/3422622. URLhttps://doi.org/10.1145/3422622. 11 Running Title for Header
-
[30]
Diffusion models beat gans on image synthesis
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, volume 34, pages 8780–8794. Curran Associates, Inc., 2021. URL https://proceedings. neurips.cc/paper_files/paper/2021/file/49ad23d1...
2021
-
[31]
A review on generative adversarial networks: Algorithms, theory, and applications.IEEE Transactions on Knowledge and Data Engineering, 35(4):3313–3332,
Jie Gui, Zhenan Sun, Yonggang Wen, Dacheng Tao, and Jieping Ye. A review on generative adversarial networks: Algorithms, theory, and applications.IEEE Transactions on Knowledge and Data Engineering, 35(4):3313–3332,
-
[32]
doi: 10.1109/TKDE.2021.3130191
-
[33]
Boyuan Ma, Xiaoyan Wei, Chuni Liu, Xiaojuan Ban, Haiyou Huang, Hao Wang, Weihua Xue, Stephen Wu, Mingfei Gao, Qing Shen, Michele Mukeshimana, Adnan Omer Abuassba, Haokai Shen, and Yanjing Su. Data augmentation in microscopic images for material data mining.npj Computational Materials, 6(1): 125, Aug 2020. ISSN 2057-3960. doi: 10.1038/s41524-020-00392-6. U...
-
[34]
Duway Nicolas Lesmes-Leon, Andreas Dengel, and Sheraz Ahmed. Systematic review of generative adversarial networks (gans) in cell microscopy: Trends, practices, and impact on image augmentation.PLOS ONE, 20(6):1–35, 06 2025. doi: 10.1371/journal.pone.0291217. URLhttps://doi.org/10.1371/journal.pone.0291217
-
[35]
Chunguang Shen, Jingxuan Zhao, Minghao Huang, Chenchong Wang, Yuqi Zhang, Wei Xu, and Shijian Zheng. Generation of micrograph-annotation pairs for steel microstructure recognition using the hybrid deep generative model in the case of an extremely small and imbalanced dataset.Materials Characterization, 217:114407, 2024. ISSN 1044-5803. doi: https://doi.or...
-
[36]
Deep generative models-assisted automated labeling for electron microscopy images segmentation, 2024
Wenhao Yuan, Bingqing Yao, Shengdong Tan, Fengqi You, and Qian He. Deep generative models-assisted automated labeling for electron microscopy images segmentation, 2024. URL https://arxiv.org/abs/2407. 19544. arXiv:2407.19544
arXiv 2024
-
[37]
High-resolution image synthesis with latent diffusion models, 2022
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022. URL https://arxiv.org/abs/2112.10752. arXiv:2112.10752
Pith/arXiv arXiv 2022
-
[38]
Amirhossein Kazerouni, Ehsan Khodapanah Aghdam, Moein Heidari, Reza Azad, Mohsen Fayyaz, Ilker Haci- haliloglu, and Dorit Merhof. Diffusion models in medical imaging: A comprehensive survey.Medical Im- age Analysis, 88:102846, 2023. ISSN 1361-8415. doi: https://doi.org/10.1016/j.media.2023.102846. URL https://www.sciencedirect.com/science/article/pii/S136...
-
[39]
Chixiang Lu, Kai Chen, Heng Qiu, Xiaojun Chen, Gu Chen, Xiaojuan Qi, and Haibo Jiang. Diffusion-based deep learning method for augmenting ultrastructural imaging and volume electron microscopy.Nature Communications, 15(1):4677, Jun 2024. ISSN 2041-1723. doi: 10.1038/s41467-024-49125-z. URL https://doi.org/10.1038/ s41467-024-49125-z
-
[40]
Riegler, and Vajira Thambawita
Roman Macháˇcek, Leila Mozaffari, Zahra Sepasdar, Sravanthi Parasa, Pål Halvorsen, Michael A. Riegler, and Vajira Thambawita. Mask-conditioned latent diffusion for generating gastrointestinal polyp images. InProceedings of the 4th ACM Workshop on Intelligent Cross-Data Analysis and Retrieval, ICDAR ’23, page 1–9, New York, NY , USA, 2023. Association for ...
-
[41]
Kevin G. Field, Xunxiang Hu, Kenneth C. Littrell, Yukinori Yamamoto, and Lance L. Snead. Radiation tolerance of neutron-irradiated model fe–cr–al alloys.Journal of Nuclear Materials, 465:746–755, 2015. ISSN 0022-3115. doi: https://doi.org/10.1016/j.jnucmat.2015.06.023. URL https://www.sciencedirect.com/science/article/ pii/S0022311515300489
-
[42]
Kevin G. Field, Samuel A. Briggs, Xunxiang Hu, Yukinori Yamamoto, Richard H. Howard, and Kumar Sridharan. Heterogeneous dislocation loop formation near grain boundaries in a neutron-irradiated commercial fecral alloy. Journal of Nuclear Materials, 483:54–61, 2017. ISSN 0022-3115. doi: https://doi.org/10.1016/j.jnucmat.2016.10
-
[43]
URLhttps://www.sciencedirect.com/science/article/pii/S0022311516306596
-
[44]
Kevin G. Field, Samuel A. Briggs, Kumar Sridharan, Yukinori Yamamoto, and Richard H. Howard. Dislocation loop formation in model fecral alloys after neutron irradiation below 1 dpa.Journal of Nuclear Materials, 495:20–26, 2017. ISSN 0022-3115. doi: https://doi.org/10.1016/j.jnucmat.2017.07.061. URL https://www. sciencedirect.com/science/article/pii/S00223...
-
[45]
Chad M. Parish, Kevin G. Field, Alicia G. Certain, and Janelle P. Wharry. Application of stem characterization for investigating radiation effects in bcc fe-based alloys.Journal of Materials Research, 30(9):1275–1289, May 2015. ISSN 2044-5326. doi: 10.1557/jmr.2015.32. URLhttps://doi.org/10.1557/jmr.2015.32. 12 Running Title for Header
-
[46]
B. Yao, D.J. Edwards, and R.J. Kurtz. Tem characterization of dislocation loops in irradiated bcc fe-based steels. Journal of Nuclear Materials, 434(1):402–410, 2013. ISSN 0022-3115. doi: https://doi.org/10.1016/j.jnucmat. 2012.12.002. URL https://www.sciencedirect.com/science/article/pii/S0022311512006587. Spe- cial Section on Spent Nuclear Fuel
-
[47]
Topacio, Najmeh Mashhadi, and S
Abolfazl Zargari, Benjamin R. Topacio, Najmeh Mashhadi, and S. Ali Shariati. Enhanced cell segmentation with limited training datasets using cycle generative adversarial networks.iScience, 27(5):109740, 2024. ISSN 2589-0042. doi: 10.1016/j.isci.2024.109740. URLhttps://doi.org/10.1016/j.isci.2024.109740
-
[48]
Few-shot image generation with diffusion models,
Jingyuan Zhu, Huimin Ma, Jiansheng Chen, and Jian Yuan. Few-shot image generation with diffusion models,
- [49]
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.