REVIEW 3 major objections 6 minor 26 references
A 3D Facial Reconstruction Evaluation Methodology: Comparing Smartphone Scans with Deep Learning Based Methods Using Geometry and Morphometry Criteria
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Smartphone TrueDepth scans beat three deep-learning 3D face reconstruction methods on both geometric and morphometric metrics.
desk verdict A genuinely useful integration of geometric morphometrics into 3D face reconstruction benchmarking, held back by an unvalidated landmarker that the morphometric results lean on. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the integration of geometric surface-deviation metrics with geometric morphometrics. Geometric error is measured by point-to-point distances via kD-tree nearest-neighbor search and by surface-to-surface deviation maps. Morphometric fidelity is quantified from 21 anatomical landmarks automatically placed by a multi-view consensus CNN (MV-CNN) trained on 125 separate SPG scans; the landmarks feed centroid size, pairwise Procrustes distances, GPA with PCA morphospaces and convex-hull IoU, Procrustes distance with permutation tests, and EDMA on 210 inter-landmark distances with sex-subgroup comparisons. The MV-CNN landmarking is the linchpin that converts raw meshes into comparable shape variables, and the entire morphometric analysis depends on its accuracy and lack of method-specific bias.
What would settle it
Manually landmark a random subset of the 82 subjects' meshes from each method by an independent expert, re-run the GPA, Procrustes-distance, and EDMA analyses on those manual landmarks, and check whether the MP > HRN3 > 3DDFA_V3 > Era3D ranking and the reported IoU and matching-distance values are reproduced; if the ranking flips or the differences shrink, the automatic landmarker was the source of the morphometric findings.
Extended reading notes
Core claim
The paper's central discovery is that, under a combined geometric-and-morphometric evaluation, smartphone-based TrueDepth scanning preserves facial morphology more faithfully than current deep-learning reconstructions from 2D images. The smartphone scans achieve the lowest point-to-point error (0.96 mm average), the lowest Procrustes distance (PD = 0.026), the highest convex-hull overlap in GPA morphospace (IoU = 0.62), and a perfect 100% top-5 EDMA matching of sex-related inter-landmark distances, while Era3D performs worst on nearly all metrics. The authors argue that these results demonstrate that their methodology provides a more comprehensive and biologically meaningful validation framework than geometry-only benchmarks, and that the ranking is consistent across independent shape-analysis tools.
Load-bearing premise
The load-bearing premise is that the automatic 21-landmark model, trained only on stereophotogrammetry scans, places landmarks equally accurately and without systematic bias on iPhone TrueDepth scans and on the three deep-learning reconstructions, so that the morphometric comparisons reflect genuine shape differences rather than landmarking artifacts.
Editorial extensions
If this is right
- Benchmarks that report only point-to-point or surface error will fail to capture biologically relevant shape loss in deep-learning reconstructions, so future validation protocols should include morphometric metrics of the kind proposed here.
- The consistent ranking of MP > HRN3 > 3DDFA_V3 > Era3D across point-to-point distance, Procrustes distance, convex-hull IoU, and EDMA matching suggests a genuine performance gradient rather than a metric-specific artifact.
- Smartphone TrueDepth scanning, despite its higher accuracy, requires a static capture process, whereas 2D-image-based methods offer more flexible acquisition and are close enough that further algorithmic improvement could make them clinically viable.
- The methodology provides a template for validating any low-cost acquisition setup against stereophotogrammetry ground truth, including other smartphone models and structured-light accessories.
Reading between the lines
- A direct test of the method's validity would be to have human experts manually place the 21 landmarks on a random subset of meshes from each method and re-run the GPA and EDMA analyses; if the MP > HRN3 > 3DDFA_V3 > Era3D ranking changes, the automatic landmarker's bias, not true shape, drove the reported morphometric results.
- Because the landmarking model was trained only on SPG meshes, its behavior on lower-resolution or topologically different meshes is the least controlled part of the pipeline; a future study could train the landmarker on a mixed set including iPhone and reconstruction meshes and compare the resulting rankings.
- The EDMA sex-dimorphism finding (MP detects eye-spacing differences like SPG, other methods do not) suggests that morphometric metrics could serve as sensitive, clinically interpretable diagnostic features for conditions with facial dysmorphology, a use the paper mentions but does not develop.
- If this methodology is applied to other deep-learning reconstruction methods (e.g., H3D-Net or SyncDreamer), it could produce a public leaderboard of morphological fidelity that would be more informative than existing geometric-error benchmarks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a methodology for evaluating low-cost 3D facial acquisition and reconstruction methods against high-end stereophotogrammetry (SPG) ground truth, combining classical geometric metrics (point-to-point distance, surface-to-surface deviation) with geometric morphometric analyses (centroid size, pairwise Procrustes distance, generalized Procrustes analysis, and Euclidean distance matrix analysis). As a case study, the authors compare iPhone TrueDepth scans (MP) and three deep-learning reconstruction methods (3DDFA_V3, HRN3, Era3D) on 82 subjects. They report that MP achieves the lowest average point-to-point error (<1 mm) and the highest morphometric similarity to SPG across most metrics, with Era3D consistently worst. The paper makes code, landmark coordinates, and the trained landmarking model publicly available.
Significance. If the findings hold, the proposed evaluation framework would provide a more biologically meaningful alternative to purely geometric benchmarks such as NoW and REALY, and the case-study ranking (MP > HRN3 > 3DDFA_V3 > Era3D) would be a useful reference for practitioners. The release of code, landmark data, and the landmarking model is a clear strength that supports reproducibility. However, the significance is currently tempered by two unresolved issues: the automatic landmarker is trained only on SPG meshes without validation on the other mesh types, and the paper lacks direct statistical comparisons between methods. These issues affect the central claim that smartphone scans are more morphometrically accurate than deep-learning reconstructions, because the morphometric results are entirely derived from the unvalidated landmarks and the ranking is based on point estimates without confidence intervals or pairwise tests.
major comments (3)
- [Section 2.3] The 21-landmark MV-CNN used for all morphometric analyses was trained exclusively on 125 SPG scans, yet it is applied without any validation to iPhone TrueDepth meshes and to 3DDFA_V3, HRN3, and Era3D reconstructions, which differ in topology, vertex density, and surface quality. Because every result in Section 3.2 (CS/PPD correlations, GPA IoU, Procrustes distance, and EDMA matching distances) is computed from these automatically placed landmarks, the observed differences between methods could reflect method-specific landmark localization bias rather than genuine shape differences. The statement in Section 3.2 that 'the only variable factor was the acquisition/reconstruction method' is not established without evidence that landmarking accuracy is consistent across the different mesh types. Please report landmark localization error on a manually landmarked subset of each method's meshes, or otherwise demonstrate that the landmarker is unbiased across the methods compared.
- [Section 3.2] The paper reports only within-method statistical tests (e.g., p<0.0001 for correlations, p<0.05 for GPA differences) and then ranks methods by point estimates such as Procrustes distance (MP 0.026, HRN3 0.033, 3DDFA_V3 0.059, Era3D 0.091) and EDMA matching-distance percentages. No confidence intervals or pairwise tests are provided, so the difference between, say, MP and HRN3 is not shown to be statistically significant. The claim that MP outperforms the deep-learning methods in morphometric accuracy would be much stronger if accompanied by bootstrap confidence intervals for PD and MD, or by permutation tests comparing the distributions of distances between methods.
- [Sections 2.2 and 2.4] Several analysis choices appear arbitrary and may affect the reported ranking: the facial region is extracted using a 100 mm sphere centered at the nose tip, and the EDMA evaluation uses the top-5 and top-10 most significant inter-landmark distances with α=0.10. No justification or sensitivity analysis is provided for these thresholds. Since the ranking (e.g., MP achieving 100% vs 40% in top-5 MD) could depend on these parameters, please justify the choices or demonstrate that the ranking is robust to reasonable variations in the sphere radius, the number of top distances, and the significance level.
minor comments (6)
- [Section 2.4] The description of the GPA permutation test states that 10,000 permutations were used, but the exact permutation scheme (what is permuted, how the test statistic is computed) is not specified. Please clarify.
- [Section 3.2, Figure 5] The text reports that all CS and PPD correlations were significant with p<0.0001, but the actual correlation coefficients and their confidence intervals are not reported in the text. Including these values would make the comparisons more interpretable.
- [Section 3.2, Figure 6] The description of the GPA morphospace analysis would benefit from a more explicit explanation of how the convex hull is constructed and how the IoU is computed, since these details are not standard in the morphometrics literature.
- [Section 2.2] The manual alignment of all low-cost reconstructions to SPG using five manually selected landmarks in MeshLab is a potential source of operator bias. No inter-operator reliability or repeatability analysis is reported for this step; consider addressing this in the revision.
- [References] Reference [25] (Paulsen et al.) is missing the publication venue; please complete the bibliographic information.
- [Throughout] There are several typos and minor wording issues: 'acquistion' in Section 3.2, 'Eras3D' in the experimental setup paragraph, 'desviation' in the caption of Figure 4, and 'Eras3D' in the acknowledgments. Please correct these.
Circularity Check
No significant circularity: the evaluation pipeline uses an external SPG ground truth, standard morphometric statistics, and a landmarker applied uniformly to all methods; the reported rankings come from measured distances and not from fitted parameters.
full rationale
The derivation chain is self-contained and not circular. Ground-truth SPG and low-cost reconstructions are independently acquired (Section 2.2). Geometric metrics are computed as point-to-point nearest-neighbour distances via kD-tree search (Section 2.4), with values reported in Table 1. Morphometric metrics are standard GPA, PD, CS/PPD correlations, and EDMA computed from landmark coordinates (Sections 2.4 and 3.2). The 21-landmark MV-CNN used for automatic landmarking (Section 2.3) is trained on a proprietary SPG dataset 'distinct from the evaluation dataset' and is then applied identically to SPG, MP, 3DDFA_V3, HRN3, and Era3D meshes; it therefore does not encode the MP-vs-DL ranking by construction. The authors' own Face3DBiomark system is included as one method under comparison, and the self-citations ([18], [26]) provide the capture tool and hyperparameter choices, not the evaluation outcome. Any concern that the landmarker may generalize unevenly across mesh topologies is a validity or domain-shift issue, not circularity, because no reported evaluation metric is algebraically equal to a training target or fitted parameter. Thus the central claim that smartphone scans achieve higher geometric and morphometric similarity to SPG is an empirical result with independent content.
Assumptions & free parameters
free parameters (5)
- Facial region extraction sphere radius =
100 mm
- EDMA top-n matching distances =
n = 5 and n = 10
- EDMA significance level alpha =
0.10
- Landmark configuration size =
21 landmarks
- GPA permutation count =
10,000
assumptions (4)
- domain assumption Stereophotogrammetry models are an accurate ground truth for facial morphology.
- domain assumption The 21 landmarks are anatomically corresponding across all acquisition/reconstruction methods.
- domain assumption Participants maintained a neutral expression and stable pose across captures.
- standard math Standard geometric morphometric assumptions (GPA, EDMA) hold for the landmark coordinates.
Cite this review
Pith. "Pith review of A 3D Facial Reconstruction Evaluation Methodology: Comparing Smartphone Scans with Deep Learning Based Methods Using Geometry and Morphometry Criteria." pith.science (2026). https://pith.science/paper/3OI2ACAP
@misc{pith2026250209425,
author = {Pith},
title = {Pith review of: A 3D Facial Reconstruction Evaluation Methodology: Comparing Smartphone Scans with Deep Learning Based Methods Using Geometry and Morphometry Criteria},
year = {2026},
howpublished = {\url{https://pith.science/paper/3OI2ACAP}},
note = {Machine review of arXiv:2502.09425}
}
read the original abstract
Three-dimensional (3D) facial shape analysis has gained interest due to its potential clinical applications. However, the high cost of advanced 3D facial acquisition systems limits their widespread use, driving the development of low-cost acquisition and reconstruction methods. This study introduces a novel evaluation methodology that goes beyond traditional geometry-based benchmarks by integrating morphometric shape analysis techniques, providing a statistical framework for assessing facial morphology preservation. As a case study, we compare smartphone-based 3D scans with state-of-the-art deep learning reconstruction methods from 2D images, using high-end stereophotogrammetry models as ground truth. This methodology enables a quantitative assessment of global and local shape differences, offering a biologically meaningful validation approach for low-cost 3D facial acquisition and reconstruction techniques.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Li, Y., Yang, X., Li, D.: The Application of Three-Dimensional Surface Imaging System in Plastic and Reconstructive Surgery. Annals of Plastic Surgery. 77, S76 (2016)
work page 2016
-
[2]
American Journal of Orthodontics and Dentofacial Orthopedics
Sarver, D.M.: Interactions of hard tissues, soft tissues, and growth over time, and their impact on orthodontic diagnosis and treatment planning. American Journal of Orthodontics and Dentofacial Orthopedics. 148, 380–386 (2015)
work page 2015
-
[3]
Starbuck, J.M., et al.: Green tea extracts containing epigallocatechin-3-gallate mod- ulate facial development in Down syndrome. Sci Rep. 11, 4715 (2021)
work page 2021
-
[4]
Aesthetic Surgery Journal Open Forum
Parsa, S., et al.: Current and Future Photography Techniques in Aesthetic Surgery. Aesthetic Surgery Journal Open Forum. 4, ojab050 (2022)
work page 2022
-
[5]
Heike, C.L., Upson, K., Stuhaug, E., Weinberg, S.M.: 3D digital stereophotogram- metry: a practical guide to facial image acquisition. Head & Face Medicine. 6, 18 (2010)
work page 2010
-
[6]
Journal of Plastic, Reconstructive & Aesthetic Surgery
Koban, K.C., et al.: Validation of two handheld devices against a non-portable three- dimensional surface scanner and assessment of potential use for intraoperative facial imaging. Journal of Plastic, Reconstructive & Aesthetic Surgery. 73, 141–148 (2020)
work page 2020
-
[7]
Li, M., et al.: Rapid automated landmarking for morphometric analysis of three- dimensional facial scans. J Anat. 230, 607–618 (2017)
work page 2017
-
[8]
In: Pertusa, A., Gallego, A.J., Sánchez, J.A., and Domingues, I
Heredia-Lidón, Á., et al.: Automated Orientation Detection of 3D Head Re- constructions from sMRI Using Multiview Orthographic Projections: An Image 12 Authors Suppressed Due to Excessive Length Classification-Based Approach. In: Pertusa, A., Gallego, A.J., Sánchez, J.A., and Domingues, I. (eds.) Pattern Recognition and Image Analysis. pp. 603–614. Spring...
work page 2023
Show all 26 references
-
[9]
Journal of Oral and Maxillofacial Surgery
Gibelli, D., Pucciarelli, V., Cappella, A., Dolci, C., Sforza, C.: Are Portable Stereophotogrammetric Devices Reliable in Facial Imaging? A Validation Study of VECTRA H1 Device. Journal of Oral and Maxillofacial Surgery. 76, 1772–1784 (2018)
2018
-
[10]
Angle Or- thod
D’Ettorre, G., et al.: A comparison between stereophotogrammetry and smart- phone structured light technology for three-dimensional face scanning. Angle Or- thod. 92, 358–363 (2022)
2022
-
[11]
Presented at the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) June 1 (2023)
Lei, B., Ren, J., Feng, M., Cui, M., Xie, X.: A Hierarchical Representation Network for Accurate and Detailed Face Reconstruction from In-The-Wild Images. Presented at the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) June 1 (2023)
2023
-
[12]
In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
Deng, Y., et al.: Accurate 3D Face Reconstruction With Weakly-Supervised Learn- ing: From Single Image to Image Set. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 285–295 (2019)
2019
-
[13]
In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Wang, Z., Zhu, X., Zhang, T., Wang, B., Lei, Z.: 3D Face Reconstruction with the Geometric Guidance of Facial Part Segmentation. In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1672–1682 (2024)
2024
-
[14]
In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
Ramon, E., et al.: H3D-Net: Few-Shot High-Fidelity 3D Head Reconstruction. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5600–5609 (2021)
2021
-
[15]
Li, P., et al.: Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention, http://arxiv.org/abs/2405.11616, (2024)
2024 arXiv
-
[16]
Liu, Y., et al.: SyncDreamer: Generating Multiview-consistent Images from a Single-view Image, http://arxiv.org/abs/2309.03453, (2024)
2024 arXiv
-
[17]
Tang, J., et al.: LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation, http://arxiv.org/abs/2402.05054, (2024)
2024 arXiv
-
[18]
et al.: Face3DBiomark: a low-cost solution to compute 3D facial biomarkers for diagnosing conditions with associated facial dysmorphologies
Heredia Lidón, Á. et al.: Face3DBiomark: a low-cost solution to compute 3D facial biomarkers for diagnosing conditions with associated facial dysmorphologies. IEEE International Symposium on Biomedical Imaging (ISBI). (2024)
2024
-
[19]
ACM Trans
Kazhdan, M., Hoppe, H.: Screened poisson surface reconstruction. ACM Trans. Graph. 32, 29:1-29:13 (2013)
2013
-
[20]
Head Face Med
Ritschl, L.M., et al.: Comparison of three-dimensional imaging of the nose using three different 3D-photography systems: an observational study. Head Face Med. 20, 7 (2024)
2024
-
[21]
Chai, Z., et al.: REALY: Rethinking the Evaluation of 3D Face Reconstruction, http://arxiv.org/abs/2203.09729, (2022)
2022 arXiv
-
[22]
Sanyal, S., et al.: Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision, http://arxiv.org/abs/1905.06817, (2019)
2019 arXiv
-
[23]
Evol Biol
Mitteroecker, P., Gunz, P.: Advances in Geometric Morphometrics. Evol Biol. 36, 235–247 (2009)
2009
-
[24]
Chapman and Hall/CRC, New York (2001)
Lele, S.R., Richtsmeier, J.T.: An Invariant Approach to Statistical Analysis of Shapes. Chapman and Hall/CRC, New York (2001)
2001
-
[25]
Paulsen, R.R., et al.: Multi-view consensus CNN for 3D facial landmark placement, (2019)
2019
-
[26]
In: Wachinger, C., Paniagua, B., Elhabian, S., Luijten, G., Title Suppressed Due to Excessive Length 13 and Egger, J
Heredia-Lidón, Á., et al.: A Critical Comparison Between Template-Based and Architecture-Reused Deep Learning Methods for Generic 3D Landmarking of Anatomical Structures. In: Wachinger, C., Paniagua, B., Elhabian, S., Luijten, G., Title Suppressed Due to Excessive Length 13 an...
2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.