Pith. sign in

REVIEW 4 major objections 6 minor 15 references

Mic-hackathon 2024: Hackathon on Machine Learning for Electron and Scanning Probe Microscopy

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read This paper reports that a two-day machine-learning hackathon for electron and scanning probe microscopy produced openly released benchmark datasets, a digital-twin microscope simulation, and a documented template for running such events.

desk verdict Honest event report with reproducible code, but the 'benchmark datasets and digital twins' framing overclaims what the appendices themselves show. read the letter →

arxiv 2506.08423 v2 pith:WWWQVDB2 submitted 2025-06-10 cond-mat.mtrl-sci cs.LGphysics.ins-det

classification cond-mat.mtrl-scics.LGphysics.ins-det
keywords hackathonmachinelearningelectronmicroscopyscanningprobebenchmarkdatasetsdigitaltwinautomatedopendata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that microscopy generates rich, well-structured data but lacks the standardized code ecosystems, benchmarks, and integration strategies that fields like genomics and X-ray crystallography already have. To help close that gap, it reports on a two-day hackathon that brought machine-learning researchers together with microscopy experts. The central outputs are benchmark datasets for scanning transmission electron microscopy, atomic force microscopy, and scanning tunneling microscopy, a digital twin of a microscope that exposes the same scripting interface as real hardware, and a repeatable organizational playbook. The claim is that these openly released resources give the community a common testing ground, so analysis workflows stop being one-off efforts and become reusable and comparable. If that holds, the field gains a concrete starting point for benchmarking and for real-time, machine-learning-driven microscope control.

What carries the argument

The load-bearing object is the digital twin microscope (DTMicroscope), a simulated microscope that reproduces the scripting interface of a real instrument, allowing participants to write and test automation code safely. Around it sits a curated dataset collection for scanning transmission electron microscopy, atomic force microscopy, and scanning tunneling microscopy, each accompanied by Jupyter notebooks that provide context and analysis instructions. Together these form the shared ground on which benchmark workflows can be built and compared. The hackathon format itself, including preparation meetings, a teaming session, supported collaboration over messaging channels, and judged submissions, is the organizational machinery the paper offers as a repeatable template.

What would settle it

Check the released repositories one year after publication for external forks, issue reports, and citations by groups unconnected to the organizing team, and look for any benchmark protocol or leaderboard built on the datasets; if no independent use or comparison metric appears, the claim that the hackathon produced community-wide benchmarks and standardized workflows is not supported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a focused, hybrid hackathon can produce the seed infrastructure that microscopy lacks: curated, permission-cleared datasets covering several imaging modes, a digital-twin microscope environment in which participants can develop and test automation code without touching physical instruments, and a documented playbook for organizing such events. The paper reports that roughly twenty project teams and eighty participants used these resources to work on segmentation, artifact removal, physics-based property inference, and automated experiment control. It treats the released datasets and digital twin as benchmark resources that can support standardized workflows and help train a machine-learning-literate microscopy workforce.

Load-bearing premise

The paper assumes that making datasets, a simulated microscope, and code openly available is enough to create benchmarks and standardized workflows, but it does not measure whether anyone outside the event actually adopts them.

Editorial extensions

If this is right

  • If the datasets and digital twin are adopted, new machine-learning workflows for microscopy can be tested against common ground truth instead of bespoke images.
  • Because the digital twin emulates the scripting interface of a real microscope, it offers a low-risk path from offline experiments to real-time, ML-agent-controlled microscope operation.
  • The documented hackathon playbook can be rerun by other communities, potentially extending the approach beyond electron and probe microscopy.
  • Researchers can benchmark segmentation, reconstruction, and artifact-removal models against the released data, which the paper identifies as a missing capability in the field.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not itself define a benchmark protocol with agreed metrics; adding one to these datasets would make model claims in the microscopy-ML literature directly comparable.
  • The digital twin could serve as a testbed for autonomous experiment agents more broadly, since it exposes the same interface as real hardware without risking instrument damage; the paper gestures at this but does not demonstrate it.
  • A measurable test of the paper's central claim is to track external use of the released code and data in the following year; independent forks, citations, or benchmark leaderboards would indicate genuine community adoption.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This manuscript reports on the organization and outcomes of the December 2024 'Mic-hackathon' on machine learning for electron and scanning probe microscopy, held in hybrid format at the University of Tennessee. It describes the preparation, logistics, datasets and simulator provided to participants, the judging process, and the 20 submitted projects, with 19 appended individual writeups. The central claim, stated in the abstract and in Section I.C, is that the hackathon 'produced benchmark datasets and digital twins of microscopes to support community growth and standardized workflows,' with all code released on GitHub and Zenodo.

Significance. If the benchmark and digital-twin claims were fully supported, this would be a genuinely useful community contribution: openly released microscopy datasets, a scriptable microscope simulator, and a documented template for running ML-microscopy hackathons. The verifiable core is credible—an event took place, 20 projects were submitted, and code repositories and presentation links are provided. The paper's main strength is its detailed organizational narrative and honest self-assessments in the appendices. However, the evidence does not currently establish the benchmark or digital-twin claims: no benchmark protocol is defined, no fidelity validation is reported for DTMicroscope, and the appendices repeatedly describe exploratory or unsuccessful outcomes rather than validated results. The significance of the paper as it stands is therefore that of a useful hackathon report and organizational template, not yet that of a benchmark-resource paper.

major comments (4)
  1. [Abstract and Section I.C] The abstract and Section I.C claim that the hackathon 'produced benchmark datasets and digital twins of microscopes,' but no benchmark protocol appears anywhere in the manuscript. Section III.C.1.b describes Google Drive datasets and DTMicroscope as community resources, yet there are no fixed task definitions, ground-truth metrics, train/test splits, or baseline results for these datasets. The appendices actively undercut the benchmark reading: Appendix 5 reports overfitting with reversed c-domain artifacts, Appendix 8 trains on a single source image and underperforms Gwyddion on PSNR for one case, and Appendix 14 states the clustering approach was 'generally inconclusive.' Please either supply a concrete benchmark protocol with quantitative evaluation, or reframe the claim to 'curated example datasets and a simulator' that are intended to support future benchmarking.
  2. [Section III.C.1.b] DTMicroscope is labeled a 'digital twin' and 'digital replica' of a microscope, but the manuscript provides no calibration, fidelity, or validation data connecting the simulator to any specific physical microscope. A shared scripting interface makes DTMicroscope a simulator; a digital twin requires demonstrated correspondence to a real instrument's behavior, including noise, drift, tip effects, or other physical responses. Please provide such validation, or revise the terminology to 'simulator' or 'simulated microscope environment' throughout the paper.
  3. [Sections III.C.3.b and Appendix 5] The first-place project 'GANder' is described in Appendix 5 as having 'struggled with overfitting, sometimes generating inaccurate predictions of reversed c-domains,' which is the opposite of a benchmark-quality result. Using this project as the headline example of the hackathon's benchmarking value is internally inconsistent. The award reflects judging criteria (innovation, technical execution, difficulty, teamwork, presentation) rather than scientific validation, so the paper should clearly separate 'winning the hackathon' from 'demonstrating a validated ML method.'
  4. [Sections I.C and III.C.3] The claim that the hackathon supports 'community growth and standardized workflows' rests entirely on self-reported participation numbers and organizer-designed awards. The paper gives no evidence of external adoption, independent reuse, or cross-team uptake of the released datasets, DTMicroscope, or project code. Because this is a load-bearing part of the abstract's claim, please either add evidence of downstream use or explicitly mark these as intended outcomes awaiting verification.
minor comments (6)
  1. [Section IV] The subsection numbering is duplicated: both 'IV.A. Image segmentation and shape analysis' and 'IV.A. Optimization of image analysis workflows' appear; these should be renumbered sequentially.
  2. [Section III.C.2] The phrase 'lecture hell' in the caption of Figure 3 appears to be a typo for 'lecture hall.'
  3. [Section I.C] The phrase 'scanning robe techniques' should be 'scanning probe techniques.'
  4. [Affiliations list] The affiliation list contains a duplicated entry '21,21b' for two authors; please clean up the affiliation numbering and the repeated institution entry.
  5. [Appendix 8, Table 1] The first row of Table 1 ('Corrupted Image 16.9 24.8 0.086 0.069') is ambiguous; the column headers need to be repeated or the row should be split by image to make the PSNR and VIF values readable.
  6. [Appendix 13] The general-intelligence polynomial model in Appendix 13 is typeset with garbled subscripts and superscripts (e.g., '𝑇+&,(𝑃%$;𝑥,𝑡&')'); please use proper math typesetting so the equation is actually legible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a descriptive hackathon report with no derivation chain whose conclusions reduce to their inputs.

full rationale

This manuscript contains no equations that define one quantity in terms of another, no fitted parameter that is later relabeled as a prediction, and no uniqueness theorem invoked to force a modeling choice. Its central claims—that the event produced datasets, a digital-twin microscope interface, code repositories, and project summaries—are descriptive statements about released artifacts, not derived results. The paper indeed contains many self-citations (e.g., refs. 5, 68, 90, 96, 99, 113–119, and the PyCroscopy/DTMicroscope ecosystem described in Section III.C.1.b), and the organizing group evaluates its own event, but none of these citations carries the weight of a derivation: no conclusion is obtained by citing the authors' prior work as a substitute for reasoning. The claimed 'benchmark datasets' are not demonstrated through benchmark protocols, metrics, or splits, and several appendices are explicitly self-critical (Appendix 5 reports GAN overfitting and reversed c-domains; Appendix 8 trains on a single source image; Appendix 14 calls the clustering approach 'generally inconclusive'). Those are concerns about support and correctness of the benchmark claim, not about circularity: the report does not define the datasets in terms of the benchmark claim, nor does it fit a parameter and then celebrate the fit as an independent prediction. Under the hard rules, no circular step can be quoted with a specific reduction, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The report's central claim is about community resources, not a mathematical or physical result, so no fitted constants or invented entities are needed. The one unstated assumption that matters is that open release of code, datasets, and a digital twin is sufficient to create benchmarks and an ecosystem; the paper provides no adoption or validation evidence for this.

assumptions (1)
  • domain assumption Releasing code, datasets, and digital twins under an open license constitutes establishing benchmarks and data analysis ecosystems.
    The abstract and Section I.C claim ecosystem creation without evidence of adoption, benchmark validation, or independent use. This unstated premise is load-bearing for the paper's central claim about community impact.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mic-hackathon 2024: Hackathon on Machine Learning for Electron and Scanning Probe Microscopy." pith.science (2026). https://pith.science/paper/WWWQVDB2

@misc{pith2026250608423,
  author       = {Pith},
  title        = {Pith review of: Mic-hackathon 2024: Hackathon on Machine Learning for Electron and Scanning Probe Microscopy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WWWQVDB2}},
  note         = {Machine review of arXiv:2506.08423}
}
read the original abstract

Microscopy is a primary source of information on materials structure and functionality at nanometer and atomic scales. The data generated is often well-structured, enriched with metadata and sample histories, though not always consistent in detail or format. The adoption of Data Management Plans (DMPs) by major funding agencies promotes preservation and access. However, deriving insights remains difficult due to the lack of standardized code ecosystems, benchmarks, and integration strategies. As a result, data usage is inefficient and analysis time is extensive. In addition to post-acquisition analysis, new APIs from major microscope manufacturers enable real-time, ML-based analytics for automated decision-making and ML-agent-controlled microscope operation. Yet, a gap remains between the ML and microscopy communities, limiting the impact of these methods on physics, materials discovery, and optimization. Hackathons help bridge this divide by fostering collaboration between ML researchers and microscopy experts. They encourage the development of novel solutions that apply ML to microscopy, while preparing a future workforce for instrumentation, materials science, and applied ML. This hackathon produced benchmark datasets and digital twins of microscopes to support community growth and standardized workflows. All related code is available at GitHub: https://github.com/KalininGroup/Mic-hackathon-2024-codes-publication/tree/1.0.0.1

Figures

Figures reproduced from arXiv: 2506.08423 by the authors.

Figure 3
Figure 3. (a) Guests arriving to the in-person meeting. (b) Lecturer detailing MSA StC award during hybrid session. (c) Representative photo of the lecture hell during the hackathon working hours. (d) Two enthusiasts touring the University of Tennessee Electron Microscopy Core after the Hackathon. Photos courtesy of Steven Spurgeon. On the day of the hackathon, Slack served as the primary communication channel, ensuring real-… view at source ↗
Figure 1
Figure 1. a) AFM image of perovskite. b) Image a) segmented by cellSAM. c) Map of interface density with regions of interest marked by black squares [PITH_FULL_IMAGE:figures/full_fig_p033_1.png] view at source ↗
Figure 2
Figure 2. interactEM workflow for AFM perovskite image [PITH_FULL_IMAGE:figures/full_fig_p033_2.png] view at source ↗
Figures from the paper (12 more)
Figure 1
Figure 1. Figure 1: Representative GAN output showing identification of ferroelastic–ferroelectric domains. (a) PFM topography channel, used as GAN input. (b) PFM amplitude channel, treated by model as ‘Ground Truth’, showing both a-domains and c–c domain walls. (c) GAN output, showing th…
Figure 2
Figure 2. Figure 2: Representative GAN output showing less ideal correlation. (a) PFM topography channel, used as GAN input. (b) PFM amplitude channel, treated by model as ‘Ground Truth’, showing both a-domains and c–c domain walls. (c) GAN output, showing both the a-domain walls as in th…
Figure 3
Figure 3. Figure 3: Generator L1 loss per iteration (light blue), and a Savitzky-Golay smoothed function generated over a 5000-step window (dark blue). ACKNOWLEDGEMENTS PbTiO3 samples were grown by Sahar Saremi and Lane W. Martin (University of California, Berkeley). R. B. acknowledges su…
Figure 1
Figure 1. Figure 1: DMD applied to HS-AFM images of the SARS-CoV-2 spike protein. WT variant (a) AFM experiments and (b) DMD reconstruction. The Omicron variant is shown in (c) Experiments and (d) DMD reconstruction. (e, f) RMSD and Rg vs. number of frames for AFM and DMD WT variant, resp…
Figure 2
Figure 2. Figure 2: Structural comparison of the spike protein using MD simulations and DMD analysis. [PITH_FULL_IMAGE:figures/full_fig_p044_2.png]
Figure 1
Figure 1. Figure 1: AFM image analysis: corrupted (left), ground truth or reference (right). [PITH_FULL_IMAGE:figures/full_fig_p049_1.png]
Figure 3
Figure 3. Figure 3: The upper panel presents NN results, showing a strong correlation between predicted and actual temperatures. The lower panel displays RF results, demonstrating high prediction accuracy as well. Insets highlight the increasing 𝑅1 and decreasing mean squared error (MSE) …
Figure 1
Figure 1. Figure 1 [PITH_FULL_IMAGE:figures/full_fig_p064_1.png]
Figure 1
Figure 1. Figure 1: Comparison of true and predicted average oxidation states for 48934 testing samples [PITH_FULL_IMAGE:figures/full_fig_p074_1.png]
Figure 2
Figure 2. Figure 2: Error histograms for each oxidation state in the test dataset, with red dashed lines indicating the RMSD. References (1) Heckbert, P. Fourier transforms and the fast Fourier transform (FFT) algorithm. Computer Graphics 1995, 2 (1995), 15-463. (2) Oyedele, A. D.; Yang, …
Figure 1
Figure 1. Figure 1: Original metallographic image and the Canny edge filter pattern. Areas such as materials discovery, metallography, geology, etc. generate large volumes of imge data sets containing depictions of the materials microstructure in multiple possible modalities. The latter c…
Figure 2
Figure 2. Figure 2: PCA components of the feature vector developed based on the LLM-generated descriptors. We note that the proposed workflow is limited in that it requires extensive human design, starting from chosen filtering, edge detection method and its parameters, patch size, and fe…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 9 canonical work pages

  1. [3]

    Micrography: GNN for Defect Detection in STEM Images Alexander Kiefer1, Nicholas Furth1 1University of Tennessee - Knoxville Abstract Micrography introduces a novel graph-based framework for analyzing electron microscope images of materials, addressing fundamental limitations of traditional pixel-based approaches. While conventional methods like convoluti...

  2. [4]

    STEM images and associated parameters for Sm-doped BFO

    Graph Neural Network Enhancement: o A Graph Autoencoder learns embeddings that preserve both molecular features and their spatial relationships o The embedding space enables identification of structural motifs and anomalies not visible in pixel space o Unsupervised clustering in the embedding space reveals regions with similar characteristics o Graph visu...

  3. [6]

    up" versus

    Unmasking biomacromolecular conformational dynamics from 2D analysis of subdomains dynamic modes and molecular kinetics Ian Addison-Smith1,3, Willy Menacho2,3 and Horacio V. Guzman2,3* 1Department of Mechanical Engineering, Universidad de Chile, Beauchef 851, Santiago, Chile 2Institut de Ciència de Materials de Barcelona, CSIC, 08193 Barcelona, Spain 3Bio...

  4. [7]

    M. O. Williams, I. G. Kevrekidis, C. W. Rowley, A data–driven approximation of the Koopman operator: Extending dynamic mode decomposition, Journal of Nonlinear Science 25, 1307–1346 (2015). 8. A. Mardt, L. Pasquali, H. Wu, F. Noé, VAMPnets for deep learning of molecular kinetics, Nature Communications 9 (2018). 9. R. Zhu, D. Canena, M. Sikora, et al., For...

  5. [8]

    Image information and visual quality

    Mitigating Tip-Induced Artifacts in AFM Images Using Autoencoder-Based Solutions Nikola L. Kolev1,2, Mikhail Petrov, Viktoriia Liu7, Sergey Ilyev5, Srikar Rairao6, Tommaso Rodani3,4 1University College London, 2London Centre for Nanotechnology, 3AREA Science Park, 4Università degli Studi di Trieste, 5Moscow Institute of Physics and Technology, 6University...

  6. [9]

    Deep Residual Learning for Image Recognition,

    GAeN: Automated Nanoparticle Detection in Microscope Images Abstract High-resolution instruments such as Transmission Electron Microscopes (TEM) are indispensable for nanoparticle analysis, offering detailed imaging of these structures. However, manual operation of these systems can be slow and inefficient, especially when processing large datasets. One o...

  7. [10]

    Department of Materials Science and Engineering, University of Illinois at Urbana−Champaign, Urbana, Illinois 61801, United States 2

    On-the-fly TEM Image Analysis for Nanoparticle Synthesis Characterization Xingzhi Wang1, Lehan Yao2, Fanzhi Su3, Pawan Vedanti4, Zhiheng Lyu1 1. Department of Materials Science and Engineering, University of Illinois at Urbana−Champaign, Urbana, Illinois 61801, United States 2. Pacific Northwest National Laboratory, Richland, WA 99354, United States 3. De...

  8. [11]

    CT4Batt: A pipeline for Analyzing X-ray Chromatography Battery Images Amir Taqieddin1 and Forrest Laskowski1 1 Technology Integration – Materials Informatics and Modeling Department, Solid Power Operating Inc, Louisville, CO 80027 Abstract. Manual detection of battery flaws using X-ray computed Tomography (CT) images is a labor-intensive and error-prone p...

Show all 15 references
  1. [12]

    Agentic Workflow for TEM Experiment Automation Xiangyu Yin1, Benjamin Fein-Ashley2, Yu-Tsun Shao3, Yi Jiang1 1Advanced Photon Source, Argonne National Laboratory, Lemont, 60439 IL, USA 2Ming Hsieh Department of Electrical and Computer Engineering, University of Southern Califo...

  2. [13]

    AI-powered Thermal Mapping in Electron-ion Trapping Experiment (EiTEx) for Single Photon Detection Vineet Kumar1*, Yogesh Paul2, Himanshu Mishra1 1Dept. of Surface and Plasma Science, Faculty of Mathematics and Physics, Charles University in Prague, Czech Republic 2Institute f...

  3. [14]

    Unsupervised Classification of Ferroelectric Domains Grace Guinan1, Addison Salvador1, Madeline Van Winkle1, Arman Ter-Petrosyan2 1 National Renewable Energy Laboratory, Golden, CO 2 Department of Materials Science and Engineering, UC Irvine Abstract A major hurdle when applyi...

  4. [15]

    Is this an adherent cell?

    MicroscopyLLM-Bench: Benchmarking LLM Capabilities for Open-Source Microscopy Datasets Adib Bazgir¹, Rama Chandra Praneeth Madugula², Yuwen Zhang¹ ¹ Department of Mechanical and Aerospace Engineering, University of Missouri-Columbia ² Department of Mechanical Engineering, New ...

  5. [16]

    High-resolution topographic imaging in AFM relies on scanning using an atomically sharp probe in either tapping or contact mode

    Removal of tip-shape induced artifacts in AFM images using deep learning Yu Liu1, Ganesh Narasimha2, Zijie Wu2 1 Department of Material Science and Engineering, University of Tennessee, Knoxville 2 Center for Nanophase Materials Sciences, Oak Ridge National Laboratory, Oak Rid...

  6. [18]

    Kalinin and Vivek Chawla Department of Materials Science and Engineering, University of Tennessee, Knoxville Figure 1

    Using LLM to enable bag-of-features segmentation of the metallographic images Sergei V. Kalinin and Vivek Chawla Department of Materials Science and Engineering, University of Tennessee, Knoxville Figure 1. Original metallographic image and the Canny edge filter pattern. Areas...

  7. [19]

    However, identifying structures from AFM images is a challenging task that relies heavily on human expertise

    Structure Discovery through Image-to-Graph Machine Learning Model Harshit Sethi, Jie Huang, Lauri Kurki Motivation Atomic Force Microscopy (AFM) with a CO functionalized tip plays a crucial role in characterizing atomic-scale nanostructures. However, identifying structures fro...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.