REVIEW 3 major objections 4 minor 36 references
A U-Net trained on human-labeled masks locates granular particles in low-gravity images with 97.7% recall, 2.7% false positives, and a mean position error of 3.7% of the particle diameter.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 19:30 UTC pith:VYVJAXM6
load-bearing objection A useful, well-executed methods paper for granular particle tracking whose headline numbers should be read as single-image point estimates, not robust statistics. the 3 major comments →
U-Net based particle localization in granular experiments: Accuracy limits and optimization
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim, on the paper's own terms, is that the particle-localization problem in these experiments reduces to a mask-design problem once a U-Net is used as the segmenter. With mask radius R=5 pixels on 38-pixel-diameter particles, anti-aliased floating-point-centered masks, and a binarization threshold chosen to maximize the F2 score, the network identifies 97.7% of particles with 2.7% false positives and a mean positional error of 1.4 pixels (3.7% of the diameter) on the test image. The paper further claims that systematic human-labeling biases (vectors of 0.2–0.6 pixels per labeler) are learned by the network and that fine-tuning on the mean coordinates of five labelers makes the
What carries the argument
The load-bearing mechanism is the pair of U-Net and carefully constructed target masks. The U-Net is a convolutional network with a contracting path, an expanding path, and skip connections; it maps each 128×128 tile to a confidence map of the same size. Training targets are binary-to-gray masks built by drawing circles at human-identified centers, and three aspects of mask construction determine performance: anti-aliased, floating-point-centered masks avoid the quantization bias of integer centers; a small mask radius (R=5 pixels) keeps neighboring masks from merging and lets the watershed postprocessing separate partially overlapping particles; and using the mean coordinates of multiple la
Load-bearing premise
The entire accuracy evaluation rests on treating human-annotated coordinates as the ground truth—the paper explicitly states that no independent ground truth is available—so the 97.7% recall and 3.7% error are measures of agreement with human labels, and if those labels are systematically wrong or the single test image is atypical, the headline numbers would not reflect true localization performance.
What would settle it
Train the identical U-Net on synthetic images with known particle centers, same overlap and illumination statistics, and compare predicted positions to the known coordinates; if the resulting error exceeds the 3.7% observed against human labels, then human-label bias rather than network architecture is the dominant error source.
If this is right
- For granular experiments in drop towers or sounding rockets, where illumination cannot be controlled and particles overlap, a U-Net pipeline can replace hand-tuned preprocessing and yield usable particle positions for trajectory reconstruction.
- The open release of code, weights, and labeled data makes this dataset a benchmark; alternative segmentation algorithms can be compared on the same images.
- Mask radius R=5 and anti-aliased masks are recommended settings for similar particle sizes: they maximize F2 score and overlap resolution simultaneously.
- Because multi-labeler averaging removes directional bias without reducing the 1.4-pixel mean error, the practical ceiling for this pipeline is set by label quality, so collecting more labelers is more productive than further architecture tuning.
- The F2 optimization with beta=2 deliberately favors recall over precision, a choice suited to trajectory linking where missed particles are more damaging than spurious detections.
Where Pith is reading between the lines
- The reported 3.7% error is an agreement-with-humans metric; the true physical localization error could be larger if human labels are biased. A synthetic image benchmark with known particle positions would separate network error from label error.
- The single held-out test image (441 tiles) leaves the 97.7%/2.7% figures without a statistical confidence interval; repeating the evaluation across several images would tell whether the performance holds for other illumination states.
- The same mask-design principles (small anti-aliased masks, labeler-ensemble ground truth) should transfer to other particle-like segmentation problems, such as bubbles, droplets, or rocks, where human annotation cost is the bottleneck.
- The paper stops at localization; feeding the U-Net outputs into a standard tracking algorithm would turn the measured 3.7% positional precision into a velocity-resolution statement, which is the eventual scientific payoff.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents a U-Net pipeline for localizing spherical granular particles in low-gravity experiments where classical thresholding fails. The authors describe the experimental setup, the failure of classical image processing, the generation of anti-aliased target masks from human-annotated particle centers, and the optimization of three hyperparameters: mask radius R, binarization threshold T, and convolution filter size f. They also analyze systematic biases among human labelers and propose fine-tuning on multi-labeler mean coordinates. On a single held-out test image labeled by six annotators, the final model achieves F2=0.976, corresponding to 97.7% recall, 2.7% false positives, and a mean positional separation of 1.4 pixels, i.e. 3.7% of the particle diameter. The paper argues that mask size controls overlap resolution, anti-aliasing enables subpixel accuracy, and human labeling biases set a measurable lower bound on attainable precision.
Significance. The pipeline is practically relevant and the paper is written clearly. The qualitative findings about mask anti-aliasing, mask radius, and the propagation of human labeler bias into the trained network are useful and well supported by the validation experiments. The open release of data, code, and weights is a real strength. However, the headline quantitative claims rest on a single test image and on human labels rather than an independent ground truth, and the test consensus includes two labelers whose annotations were used for training. If the authors extend the evaluation to a multi-image test set and make the consensus analysis independent of training labelers, this work could be a valuable benchmark for granular particle tracking. In its current form, the central accuracy claims are not yet statistically established.
major comments (3)
- [Section 5.4, Table 1] The headline performance (F2=0.976, 97.7% recall, 2.7% false positives, 3.7% diameter mean error) is computed from exactly one held-out test image. No confidence intervals, per-image breakdown, or repeated-initialization statistics are reported for this final model. The authors’ own validation data (Figs. 13–14) show non-negligible scatter: at R=5, f=3 the mean |s| ranges roughly 1.9–2.2 pixels and F2 varies across random initializations. A single test image cannot support the general claim that the pipeline provides precise, reliable particle positions for the experimental setup. Please report per-image and per-seed distributions on a multi-image test set, or explicitly restrict the quantitative claims to the demonstrated image.
- [Section 3.3, Section 4.3] The paper correctly states in Section 3.3 that there is no independent ground truth, and Section 4.3 shows that human labelers have systematic biases. Consequently, the “accuracy of 3.7% of the particle diameter” and the statement that the constant 1.4 pixels “suggests we are approaching the maximal accuracy achievable” are measures of agreement with human labels, not physical localization accuracy. Since the network is trained to reproduce human labels, a systematic labeler bias would be learned and would not appear as an error in this metric. Please re-frame these numbers as human-consensus agreement throughout, and consider a synthetic-image or calibration validation with known ground truth.
- [Section 4.3, Section 5.4] The test-image consensus is the average of six labelers, including Alice and Bob, the two labelers who generated the base training data. Figure 9 demonstrates that labeler biases are systematic and, for Alice, consistent across images. Because all evaluation metrics are computed against this consensus, the test agreement can be partially inflated by the network reproducing biases of Alice and Bob that are also present in the test labels. Please report the test metrics against the consensus of the four labelers not involved in training (Eve, Victor, Ivan, Judy), or at least provide per-labeler agreement, to rule out this effect.
minor comments (4)
- [Throughout] Typographical errors: Table 1 caption/text contains “1 for and testing”; Fig. 6 caption has “ImgageJ”; §3.2 has “Eucliedan Distance transform”; §4.2 has “betweeen”; footnote [34] contains the broken phrase “the are anti-aliased masks.”
- [Section 3.3] The true-positive matching threshold is 20 pixels, about half a particle diameter. The recall and false-positive rates are therefore insensitive to localization errors of up to 20 pixels. This is acceptable because the mean |s| metric is reported separately, but the interpretation of F2 should state this explicitly.
- [Section 6] The conclusion that insensitivity to hyperparameters “indicates that all information contained in the images has been successfully extracted” is too strong. The reported robustness only demonstrates stability within the explored hyperparameter ranges, not exhaustive information extraction.
- [Declarations] The paper says code, weights, and data “will be published” rather than providing a link or DOI. To make the reproducibility claim verifiable, a persistent link should be included in the final version.
Circularity Check
No load-bearing circularity; held-out test evaluation is standard, with the explicit caveat that ground truth is human consensus.
full rationale
The claimed derivation is a standard supervised-learning pipeline. Training uses 20 images with human-labeled masks; validation uses 7 disjoint images to choose mask radius R, cutoff T, and filter size f; the final metrics are computed on a single held-out test image (Table 1) labeled by 6 people and never used for training or fine-tuning. The 97.7% recall, 2.7% false positives, and 1.4 px mean error are therefore direct held-out predictions, not fitted values. The paper explicitly flags its main limitation: 'we do not have access to an independent ground truth' (Sec. 3.3), so all accuracy numbers measure agreement with human annotation, not physical localization error. The statement that 'human labeling biases set a measurable lower bound on achievable precision' is an interpretation of the metric |s|, which is defined against manual labels; it does not feed back into the reported test scores. The self-citations ([10] for the experimental setup, [22] for the tiling scheme) are not load-bearing: the tiling choice is standard and the central U-Net result is evaluated independently. The single-test-image evaluation and lack of confidence intervals are a support gap, not a circular step. Hence no constructed equivalence between inputs and predictions.
Axiom & Free-Parameter Ledger
free parameters (3)
- mask radius R =
5 pixels (optimal)
- binarization threshold T =
Chosen per R to maximize F2 (Fig. 14)
- convolution filter size f =
3x3
axioms (4)
- domain assumption Human-labeled particle centers are an acceptable proxy for true particle positions.
- domain assumption The U-Net architecture is capable of learning the mapping from images to particle-center confidence maps.
- domain assumption Particles are approximately circular and of known diameter D≈38 pixels.
- domain assumption Image tiles of 128×128 with 50% overlap preserve enough context for the network.
read the original abstract
Identifying the positions of granular particles from experimental images is often complicated by their partial overlap in two dimensional projections. Uneven backgrounds and inhomogeneous illuminations can add to the challenge. Conventional image-processing methods are often unable to analyze such images. We show that a deep neural network with an U-Net architecture can provide precise particle positions with a high detection rate. For our challenging test image the network correctly identifies 97.7\% of the particles while only creating 2.7 \% of false positives. The training of the U-Net requires a number of target images where the position of all particles have been identified by humans. Those positions are then indicated in the target images by setting a small number of mask pixels to white in an otherwise black image. We demonstrate that the design of these masks critically determines performance: mask size controls the resolution of overlapping particles, anti-aliased masks enable subpixel accuracy, and systematic human labeling biases set a measurable lower bound on achievable precision. Our final network achieves an accuracy of the particle coordinate of 3.7\% of the particle diameter.
Reference graph
Works this paper leans on
-
[1]
Falcon, E., Wunenburger, R., ´Evesque, P., Fauve, S., Chabot, C., Garrabos, Y., Beysens, D.: Cluster formation in a granular medium fluidized by vibrations in low gravity. Phys. Rev. Lett.83, 440–443 (1999) https://doi. org/10.1103/PhysRevLett.83.440
-
[2]
Sack, A., Heckel, M., Kollmer, J.E., Zimber, F., P¨ oschel, T.: Energy dissipation in driven granular matter in the absence of gravity. Phys. Rev. Lett.111, 018001 (2013) https: //doi.org/10.1103/PhysRevLett.111.018001
-
[3]
EPL (Europhysics Letters)103, 64004 (2013) https://doi.org/10.1209/0295-5075/103/ 64004 11
Falcon, E., Bacri, J.-C., Laroche, C.: Equation of state of a granular gas homo- geneously driven by particle rotations. EPL (Europhysics Letters)103, 64004 (2013) https://doi.org/10.1209/0295-5075/103/ 64004 11
-
[4]
Harth, K., Trittel, T., Wegner, S., Stannarius, R.: Free cooling of a granular gas of rod- like particles in microgravity. Phys. Rev. Lett. 120, 214301 (2018) https://doi.org/10.1103/ PhysRevLett.120.214301
2018
-
[5]
Yu, P., Schr¨ oter, M., Sperl, M.: Velocity distribution of a homogeneously cooling gran- ular gas. Phys. Rev. Lett.124, 208007 (2020) https://doi.org/10.1103/PhysRevLett.124. 208007
-
[6]
Icarus 360, 114307 (2021) https://doi.org/10.1016/ j.icarus.2021.114307
Schneider, N., Musiolik, G., Kollmer, J.E., Steinpilz, T., Kruss, M., Jungmann, F., Demirci, T., Teiser, J., Wurm, G.: Experi- mental study of clusters in dense granular gas and implications for the particle stop- ping time in protoplanetary disks. Icarus 360, 114307 (2021) https://doi.org/10.1016/ j.icarus.2021.114307
arXiv 2021
-
[7]
npj Micro- gravity8(1), 11 (2022) https://doi.org/10
Pitikaris, S., Bartz, P., Yu, P., Cristofore- tti, S., Sperl, M.: Granular cooling of ellip- soidal particles in microgravity. npj Micro- gravity8(1), 11 (2022) https://doi.org/10. 1038/s41526-022-00196-6
2022
-
[8]
npj Micrograv- ity10, 36 (2024) https://doi.org/10.1038/ s41526-024-00369-5
Puzyrev, D., Trittel, T., Harth, K., Stan- narius, R.: Cooling of a granular gas mixture in microgravity. npj Micrograv- ity10, 36 (2024) https://doi.org/10.1038/ s41526-024-00369-5
2024
-
[9]
Communications Physics7, 425 (2024) https://doi.org/10
Cheng, K., Hou, M., Sun, W., Qiao, Z., Li, X., Lai, C., Yuan, J., Li, T., Ye, F., Chen, K., Yang, M.: Unraveling the role of gravity in shaping intruder dynamics within vibrated granular media. Communications Physics7, 425 (2024) https://doi.org/10. 1038/s42005-024-01927-9
2024
-
[10]
Review of Scientific Instru- ments90(5), 054501 (2019) https://doi.org/ 10.1063/1.5085319
Yu, P., St¨ ark, E., Blochberger, G., Kaplik, M., Offermann, M., Tran, D., Adachi, M., Sperl, M.: Magnetically excited granular mat- ter in low gravity. Review of Scientific Instru- ments90(5), 054501 (2019) https://doi.org/ 10.1063/1.5085319
-
[11]
Microgravity Science and Technology32, 897–906 (2020) https://doi.org/10.1007/ s12217-020-09800-4
Puzyrev, D., Harth, K., Trittel, T., Stannarius, R.: Machine learning for 3d particle tracking in granular gases. Microgravity Science and Technology32, 897–906 (2020) https://doi.org/10.1007/ s12217-020-09800-4
2020
-
[12]
Pro- ceedings of the National Academy of Sciences 115, 9026–9031 (2018) https://doi.org/10
Newby, J.M., Schaefer, A.M., Lee, P.T., For- est, M.G., Lai, S.K.: Convolutional neural networks automate detection for tracking of submicron-scale particles in 2d and 3D. Pro- ceedings of the National Academy of Sciences 115, 9026–9031 (2018) https://doi.org/10. 1073/pnas.1804420115
2018
-
[13]
Applied Physics Reviews8(2021) https:// doi.org/10.1063/5.0034891
Midtvedt, B., Helgadottir, S., Argun, A., Pineda, J., Midtvedt, D., Volpe, G.: Quanti- tative digital microscopy with deep learning. Applied Physics Reviews8(2021) https:// doi.org/10.1063/5.0034891 . 011310
-
[14]
Nature Methods 18, 100–106 (2021) https://doi.org/10.1038/ s41592-020-01018-x
Stringer, C., Wang, T., Michaelos, M., Pachi- tariu, M.: Cellpose: a generalist algorithm for cellular segmentation. Nature Methods 18, 100–106 (2021) https://doi.org/10.1038/ s41592-020-01018-x
2021
-
[15]
Journal of Innovative Optical Health Sciences15(2022) https://doi.org/10.1142/s1793545822500316
Liu, Z., Zhang, H., Jin, L., Chen, J., Nedzved, A., Ablameyko, S., Ma, Q., Yu, J., Xu, Y.: U-net-based deep learning for tracking and quantitative analysis of intracellular vesicles in time-lapse microscopy images. Journal of Innovative Optical Health Sciences15(2022) https://doi.org/10.1142/s1793545822500316
-
[16]
In: The IEEE International Symposium on Biomed- ical Imaging Challenges (ISBIC) (2022)
Weigert, M., Schmidt, U.: Nuclei instance segmentation and classification in histopathology images with stardist. In: The IEEE International Symposium on Biomed- ical Imaging Challenges (ISBIC) (2022). https://doi.org/10.1109/ISBIC56247.2022. 9854534
arXiv 2022
-
[17]
Jour- nal of Open Source Software10, 5986 (2025) https://doi.org/10.21105/joss.05986
Niemann, A., Puzyrev, D., Stannarius, R.: ParticleTracking: A GUI and library for par- ticle tracking on stereo camera images. Jour- nal of Open Source Software10, 5986 (2025) https://doi.org/10.21105/joss.05986
-
[18]
In: Proceedings of the IEEE International Conference on Computer Vision, pp
He, K., Gkioxari, G., Doll´ ar, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)
2017
-
[19]
Sci- entific Reports14, 14281 (2024) https://doi
Dillavou, S., Hanlan, J.M., Chieco, A.T., Xiao, H., Fulco, S., Turner, K.T., Durian, 12 D.J.: Bellybutton: accessible and customiz- able deep-learning image segmentation. Sci- entific Reports14, 14281 (2024) https://doi. org/10.1038/s41598-024-63906-y
-
[20]
In: Medical Image Com- puting and Computer-Assisted Intervention– MICCAI 2015: 18th International Confer- ence, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp
Ronneberger, O., Fischer, P., Brox, T.: U- net: Convolutional networks for biomedical image segmentation. In: Medical Image Com- puting and Computer-Assisted Intervention– MICCAI 2015: 18th International Confer- ence, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234–241 (2015). Springer
2015
-
[21]
Journal of Industrial and Management Opti- mization19, 5331–5352 (2023) https://doi
Liang, J., Liu, X., Chen, T., Pan, C., Xu, C.: Particle identification in particle tracking velocimetry using two-stage neural networks. Journal of Industrial and Management Opti- mization19, 5331–5352 (2023) https://doi. org/10.3934/jimo.2022175
-
[22]
Granu- lar Matter24, 65 (2022) https://doi.org/10
Sanvitale, N., Gheller, C., Bowman, E.: Deep learning assisted particle identification in photoelastic images of granular flows. Granu- lar Matter24, 65 (2022) https://doi.org/10. 1007/s10035-022-01222-w
2022
-
[23]
Optics Express28, 2987–2999 (2020) https://doi.org/10.1364/OE.379480
Shao, S., Mallery, K., Kumar, S.S., Hong, J.: Machine learning holography for 3D particle field imaging. Optics Express28, 2987–2999 (2020) https://doi.org/10.1364/OE.379480
-
[24]
Granular Mat- ter27, 62 (2025) https://doi.org/10.1007/ s10035-025-01542-7
Yu, J., Meng, L., Ren, S., Song, X., Liang, H., Cao, J., Xue, Y., Zhou, W.: A method for identifying fragmentation of open-pit min- ing blasting based on a new hybrid con- volutional neural network. Granular Mat- ter27, 62 (2025) https://doi.org/10.1007/ s10035-025-01542-7
2025
-
[25]
ZARM Drop Tower Operation and Service Company, (2023)
Cornelius, M.: ZARM Drop Tower Bremen User Manual. ZARM Drop Tower Operation and Service Company, (2023). ZARM Drop Tower Operation and Service Company
2023
-
[26]
Kapoor, S., Narayanan, A.: Leakage and the Reproducibility Crisis in ML-based Science. arXiv. arXiv:2207.07048 [cs, stat] (2022). https://doi.org/10.48550/arXiv.2207.07048
-
[27]
https://github.com/bnsreenu/ python for microscopists/blob/master/ 074-DefiningU-netinPythonusingKeras.py
Bhattiprolu, S.: Defining U-net in Python using Keras. https://github.com/bnsreenu/ python for microscopists/blob/master/ 074-DefiningU-netinPythonusingKeras.py. Last accessed: Jan 31, 2020 (2020)
2020
-
[28]
Science313, 504–507 (2006) https: //doi.org/10.1126/science.1127647
Hinton, G.E., Salakhutdinov, R.R.: Reduc- ing the Dimensionality of Data with Neural Networks. Science313, 504–507 (2006) https: //doi.org/10.1126/science.1127647
-
[29]
Lin, T.-Y., Goyal, P., Girshick, R., He, K., Doll´ ar, P.: Focal Loss for Dense Object Detec- tion. arXiv. arXiv:1708.02002 [cs] (2018). https://doi.org/10.48550/arXiv.1708.02002
-
[30]
Sudre, C.H., Li, W., Vercauteren, T., Ourselin, S., Jorge Cardoso, M.: Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In: Car- doso, M.J., Arbel, T., Carneiro, G., Syeda- Mahmood, T., Tavares, J.M.R.S., Moradi, M., Bradley, A., Greenspan, H., Papa, J.P., Madabhushi, A., Nascimento, J.C., Cardoso, J.S., Belag...
2017
-
[31]
BMC Bioinformatics18, 529 (2017) https://doi
Rueden, C.T., Schindelin, J., Hiner, M.C., DeZonia, B.E., Walter, A.E., Arena, E.T., Eliceiri, K.W.: Imagej2: Imagej for the next generation of scientific image data. BMC Bioinformatics18, 529 (2017) https://doi. org/10.1186/s12859-017-1934-z
-
[32]
International Workshop on Image Processing: Real-time Edge and Motion Detection/Estimation, 12–21 (1979)
Beucher S, L.C.: Use of watersheds in contour detection. International Workshop on Image Processing: Real-time Edge and Motion Detection/Estimation, 12–21 (1979)
1979
-
[33]
https://github.com/ HumanSignal/label-studio
Tkachenko, M., Malyuk, M., Holmanyuk, A., Liubimov, N.: Label Studio: Data labeling software (2020-2025). https://github.com/ HumanSignal/label-studio
2020
-
[34]
Together with rounding by floor, this would with a 50% probability change the position of those particles coming from the ’snapping region’ to the next smaller integer
This effect could be counteracted by adding a small random perturbation−0.01< ϵ <0.01 to every particle position that happens to be an integer. Together with rounding by floor, this would with a 50% probability change the position of those particles coming from the ’snapping region’ to the next smaller integer. However, a better way to solve the problem 1...
-
[35]
In: Proceedings of the Second Inter- national Conference on Knowledge Discovery and Data Mining (KDD-96), pp
Ester, M., Kriegel, H.-P., Sander, J., Xu, X.: A density-based algorithm for discover- ing clusters in large spatial databases with noise. In: Proceedings of the Second Inter- national Conference on Knowledge Discovery and Data Mining (KDD-96), pp. 226–231 (1996)
1996
-
[36]
https://creativecommons.org/licenses/by- sa/4.0/ 14
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.