Pith. sign in

REVIEW 4 major objections 7 minor 2 cited by

3D Convolutional Neural Networks Image Registration Based on Efficient Supervised Learning from Artificial Deformations

T0 review · 4 major / 7 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Synthetic deformations alone train a CNN to match CT registration

desk verdict Solid extension of the authors' earlier synthetic-DVF idea with a fair independent evaluation; the central generalization claim needs one more control experiment before being taken as a predictive criterion. read the letter →

arxiv 1908.10235 v1 pith:Z2TLX3CR submitted 2019-08-27 eess.IV cs.CVcs.LG

Hessam Sokooti , Bob de Vos , Floris Berendsen , Mohsen Ghafoorian , Sahar Yousefi , Boudewijn P.F. Lelieveldt , Ivana Isgum , Marius Staring This is my paper · ORCID
classification eess.IVcs.CVcs.LG
keywords medicalimageregistrationnonrigidconvolutionalneuralnetworkartificialdisplacementvectorfieldschestCTrespiratorymotionsimulationsupervisedlearningmulti-stage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a convolutional network can learn deformable image registration without any manually labeled ground-truth deformations. It answers yes for chest CT: by generating large numbers of artificial displacement fields, including simulated respiratory motion, and training a multi-stage 3D CNN to predict them voxel-by-voxel, the resulting system, RegNet, matches conventional B-spline registration on real scans. On the SPREAD test set it reaches 2.32 ± 5.33 mm target registration error against 2.21 ± 5.86 mm for B-spline registration; on DIR-Lab-4DCT it reaches 1.86 ± 2.12 mm, behind the sliding-motion method at 1.36 ± 1.01 mm but ahead of other CNN approaches. The point is that densely supervised training can be obtained from synthetic deformations rather than from expensive manual annotation.

What carries the argument

The load-bearing object is the artificial training-pair generator: it takes a real moving image, applies a synthetic displacement field drawn from 14 basis types (5 single-frequency, 4 mixed-frequency, 4 respiratory, and 1 identity), adds intensity changes, and produces a fixed image with a known dense ground-truth displacement field. This turns registration into voxel-wise supervised regression without manual labels. The inference side is a three-stage RegNet pipeline in which a coarse U-Net at quarter resolution, then a refined U-Net-advanced at half resolution, then one at full resolution compose their predicted transformations, with maximum synthetic displacement increasing from 7 mm at full resolution to 20 mm at coarse resolution to extend capture range.

What would settle it

Train the U4-Uadv2-Uadv1 pipeline under the exact protocol but withhold the respiratory-motion category and test on DIR-Lab-4DCT: the paper reports 2.70 ± 4.39 mm for single-plus-mixed frequency and 1.86 ± 2.12 mm once respiratory motion is added, so if that gap disappears or reverses, the claim that simulated respiratory motion is what makes synthetic training work would be refuted. A second check is to run the trained network on a new inhale-exhale chest CT set whose diaphragmatic motion exceeds the simulated maximum; failure there would show the capture range is set by the synthetic distribution rather than by the network.

Watch

Extended reading notes

Core claim

The central claim is that sufficiently realistic artificial deformations are an adequate training signal for nonrigid registration on real data. The paper constructs this signal by combining three families of synthetic displacement fields, namely single-frequency, mixed-frequency, and simulated respiratory motion, plus intensity models such as a mass-preserving sponge adjustment and Gaussian noise. It then trains a coarse-to-fine pipeline of three 3D CNNs, a U-Net at quarter resolution followed by U-Net-advanced networks at half and full resolution, using a Huber loss with bending-energy regularization. On independent test sets the network generalizes: it matches B-spline registration on SPREAD and, once respiratory motion is added to the training distribution, improves from 2.70 ± 4.39 mm to 1.86 ± 2.12 mm on DIR-Lab-4DCT. The paper states its own conclusion as showing that training with sufficiently realistic artificially generated displacement fields can yield accurate registration results even in real cases.

Load-bearing premise

The artificial displacement fields and intensity changes used at training time are representative enough of real chest-CT deformation and appearance that a network trained only on them generalizes to unseen real scans.

Editorial extensions

If this is right

  • Training data for deformable registration can be manufactured at scale from unlabeled scans, removing the need for landmark or segmentation gold standards.
  • On inhale-exhale chest CT, adding the respiratory-motion category is what bridges the gap to real scans, moving target registration error from 2.70 mm to 1.86 mm on DIR-Lab-4DCT.
  • A two-stage version runs in about 2.2 s and reaches nearly the same accuracy as the three-stage version, so a speed-accuracy trade-off is available in practice.
  • The same network design and artificial-generation recipe, minus respiratory motion, are proposed by the paper as applicable to other modalities such as brain MRI.
  • The gap to the sliding-motion method on DIR-Lab-4DCT indicates where future gains lie: the paper points to adding sliding-motion simulation and rib-aware rigidity to the generated deformations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment is to replace the hand-designed respiratory model with a deformation generator learned from unpaired real inhale-exhale scans, which would test whether the remaining gap to the sliding-motion baseline is a realism gap or an architecture gap.
  • The DIR-Lab-COPDgene result suggests the next bottleneck is appearance change rather than deformation; adding random intensity occlusions to training pairs should measurably improve those cases.
  • Because the multi-stage design composes coarse and fine predictions, its capture range is set by the maximum synthetic displacement at the coarsest stage, so simply raising that maximum without changing the architecture should extend registration to larger breathing excursions.
  • For future synthetic-training methods on chest CT, the two decisive comparisons are B-spline registration on SPREAD and sliding-motion registration on DIR-Lab-4DCT, since those are the baselines the sufficiency claim has to match or beat.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes RegNet, a supervised multi-stage 3D CNN for nonrigid chest CT registration, trained exclusively on artificially generated displacement vector fields (DVFs) and synthetic intensity changes. Three architectures (U-Net, Multi-View, U-Net-advanced) are compared, and a coarse-to-fine three-stage pipeline is selected. Training DVFs are generated from single-frequency, mixed-frequency, and simulated respiratory-motion models plus identity, with sponge and Gaussian-noise intensity models. The selected pipeline is evaluated on SPREAD test cases and independently on DIR-Lab-4DCT, with TRE 2.32±5.33 mm and 1.86±2.12 mm respectively, and is compared with elastix B-spline, a sliding-motion conventional method, and three learning-based methods. The paper also reports DIR-Lab-COPDgene results on training/validation data showing substantially higher TRE. The code is publicly available.

Significance. If the result holds, it is a notable demonstration that a CNN can learn deformable registration from synthetic deformations alone, without manual gold-standard DVFs, and still generalize to real inhale-exhale chest CT. The positive evidence is strengthened by the independent DIR-Lab-4DCT test, the reporting of folding rates, the use of statistical testing, and the release of code. The main limitation is that the central 'sufficient realism' premise is not directly validated: the failure on DIR-Lab-COPDgene is explained post hoc, and the contribution of the respiratory-motion category was not tested on that dataset. With that test added or the claim appropriately scoped, the result would be a solid contribution to supervised and synthetic-data-based medical image registration.

major comments (4)
  1. [Section IV, Tables II and IV] The paper's central claim is that training with 'sufficiently realistic artificially generated displacement fields' yields accurate results in real cases, but the only quantitative evidence for the decisive role of realism is the improvement from 2.70±4.39 mm ('S+M') to 1.86±2.12 mm ('S+M+R') on DIR-Lab-4DCT (Table IV). The negative DIR-Lab-COPDgene result in Table II (e.g., 8.07±7.65 mm for U4-Uadv2-Uadv1) was obtained with training categories 'S' and 'M' only (Section III-D1), even though COPDgene is also an inhale-exhale chest CT dataset. The Discussion's explanation that 'sufficient realism was not added' is therefore circular in the absence of an ablation that adds the respiratory-motion category to the COPDgene training; without such an experiment, the manuscript cannot distinguish an intensity-model failure from a deformation-model failure, and the scope of the central claim remains unclear. Please add the missing condition (e.g., train with S+M+R on COPDgene) or explicitly restrict the conclusion to scenarios covered by the R category.
  2. [Table II and Section III-D1] The sentence 'A statistically significant difference (with p<0.05) between U4-Uadv2-Uadv1 and all single stage and two stages combination can be observed' is not consistent with the table: MV4-MV2-MV1, U4-MV2-MV1, and Uadv4-Uadv2-Uadv1 are three-stage combinations that have no dagger in the SPREAD column, and all listed two-stage combinations do have daggers. This makes the architecture-selection claim and the table's legend contradictory; please correct the claim or the annotation.
  3. [Table III and Section III-D2] The statement that 'in most cases there is no significant difference between B-spline registration and RegNet trained using "S" or trained using "S+M"' conflicts with the test-set 'Total' row, where all RegNet variants, including S (2.32±5.33) and S+M (2.39±5.64), carry the dagger marker, indicating a significant difference from B-spline (2.21±5.86). The per-case non-significance may reflect the small number of test cases, but the summary should be reconciled with the statistical test results.
  4. [Section III-D and Tables III-IV] All TRE values are reported as mean ± standard deviation over landmarks only; no confidence intervals or repeated training runs are shown. Because the synthetic DVF generation, data sampling, and network initialization are stochastic, the observed differences between RegNet variants and between RegNet and B-spline could be within run-to-run variability. Please report the variability across at least a few training seeds (or across synthetic-data draws) for the main comparisons, and clarify whether the Wilcoxon tests are performed over landmarks, over cases, or over pairs.
minor comments (7)
  1. [Section II-B] The activation name 'ELu' should be 'ELU'.
  2. [Table III caption] The phrase 'where † indicates a statistically significant difference' is duplicated in the caption; remove the repetition.
  3. [Section III-D2] The entry 'Eppenhof and Pluim (2018)-DIR' is not clearly distinguished from 'Eppenhof and Pluim (2018)' in the reference list; add a note explaining how the DIR variant was trained.
  4. [Section II-C1 and Table I] The 'identity' category is listed as an artificial DVF category, but Table I does not specify its settings or how many identity pairs are generated per stage; please clarify this.
  5. [Section III-C1] The sentence 'In all evaluations, images are multiplied with the lung masks' should specify whether the mask is applied to the network inputs, to the TRE computation, or to both.
  6. [Section II-C2, Eq. (1)] The Jacobian determinant J in Eq. (1) is not defined; please define J explicitly and state whether the forward or inverse transformation is used.
  7. [Figure 1] The composition order T(x)=Ts1(Ts2(Ts4(x))) is clear in the text, but the block labels (RegNet4, RegNet2, RegNet1) may confuse readers because the diagram order is opposite to the functional composition; consider adding arrows or a caption note.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RegNet is validated against independent manual landmarks on real images, and the synthetic deformation generator is not fitted to those test targets.

full rationale

The paper's derivation chain is: hand-designed artificial DVF and intensity models (Section II-C, Table I) generate training pairs; a multi-stage CNN is trained with Huber loss against those artificial ground-truth DVFs (Section II-D); the trained pipeline is then applied to real chest CT pairs and evaluated by target registration error against manually annotated landmarks (Section III-B, Eq. 4). No parameter of the DVF generator or intensity model is fitted to the test landmarks or to any real DVF; the SPREAD and DIR-Lab-4DCT test sets are independent of training (Section III-C1). The self-citations to Sokooti et al. (2017) for artificial DVF generation and multi-view architecture are historical and descriptive: the current paper contains the full generation scheme and parameter settings, and no correctness or uniqueness theorem is imported from the authors' prior work. The Discussion's statement that 'sufficient realism was not added' for COPDgene is a post-hoc attribution and is underdetermined because that experiment also omitted the respiratory-motion category; this is a scope limitation, not a circular reduction, and it does not undermine the externally measured positive results.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. The main free parameters are the hand-designed DVF generation settings and training hyperparameters. The domain assumptions concern the representativeness of synthetic training data, the validity of landmark TRE as ground truth, and the coarse-to-fine composition model. The unstated gamma value is a notable omission.

free parameters (7)
  • theta per stage = 7 mm (stage 1), 15 mm (stage 2), 20 mm (stage 4)
    Maximum artificial displacement along each axis, chosen to match the expected displacement magnitudes in the target chest CT data. Not optimized on the test set, but hand-selected per stage.
  • DVF generation settings (s and sigma_B) = Table I: 5 single-frequency, 4 mixed-frequency, 4 respiratory-motion settings
    B-spline grid spacing and Gaussian smoothing widths control the spatial frequency content of the synthetic DVFs. These values are hand-chosen to cover a plausible range of deformations.
  • sigma_N (Gaussian noise intensity) = 5 (and 3 for initial single-frequency deformations)
    Added to the deformed images to simulate realistic acquisition noise; values in Section II-C2.
  • gamma (bending energy weight) = not reported
    Regularization weight in Eq. 2. The paper never states its value, which is a reproducibility gap.
  • Number of artificial training pairs per image per stage = 70 (stage 4), 42 (stage 2), 28 (stage 1)
    Sets the size of the training dataset; chosen as 5x, 3x, and 2x the 14 basis DVF types.
  • Patch sampling balance bins = Stage 4: [0,1.5), [1.5,8), [8,20) mm; Stage 2: [0,1.5), [1.5,4), [4,15) mm; Stage 1: [0,2), [2,7) mm
    Equal sampling of patches by displacement magnitude to balance the training distribution.
  • Learning rate and optimizer = Adam, 0.001
    Standard optimization hyperparameter from Section II-D.
assumptions (4)
  • domain assumption The proposed artificial DVF and intensity models are representative enough of real chest CT deformations and appearance changes to generalize to real images.
    The method's success depends on training distribution matching the test distribution. The Discussion admits sub-optimal performance on DIR-Lab-COPDgene when realism is insufficient (Section IV).
  • domain assumption Landmark-based TRE is a valid accuracy measure for nonrigid registration.
    TRE is used for all quantitative evaluation. The paper acknowledges landmark gold standards can be sparse or uncertain (Section I) and excludes SPREAD cases 12 and 19 for this reason.
  • domain assumption The coarse-to-fine transformation composition T(x) = Ts1(Ts2(Ts4(x))) is a valid model for the deformations in the data.
    Used in Section II-A as the system output. Standard in registration, but the paper does not compare the composition model against a monolithic DVF prediction.
  • domain assumption The sponge intensity model IF = IF_clean * J_T^-1 reasonably simulates lung tissue mass preservation.
    Equation 1; one of the two intensity models, taken from prior work (Staring et al., 2014). Its validity for the test CT data is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 3D Convolutional Neural Networks Image Registration Based on Efficient Supervised Learning from Artificial Deformations." pith.science (2026). https://pith.science/paper/Z2TLX3CR

@misc{pith2026190810235,
  author       = {Pith},
  title        = {Pith review of: 3D Convolutional Neural Networks Image Registration Based on Efficient Supervised Learning from Artificial Deformations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z2TLX3CR}},
  note         = {Machine review of arXiv:1908.10235}
}
abstract

We propose a supervised nonrigid image registration method, trained using artificial displacement vector fields (DVF), for which we propose and compare three network architectures. The artificial DVFs allow training in a fully supervised and voxel-wise dense manner, but without the cost usually associated with the creation of densely labeled data. We propose a scheme to artificially generate DVFs, and for chest CT registration augment these with simulated respiratory motion. The proposed architectures are embedded in a multi-stage approach, to increase the capture range of the proposed networks in order to more accurately predict larger displacements. The proposed method, RegNet, is evaluated on multiple databases of chest CT scans and achieved a target registration error of 2.32 $\pm$ 5.33 mm and 1.86 $\pm$ 2.12 mm on SPREAD and DIR-Lab-4DCT studies, respectively. The average inference time of RegNet with two stages is about 2.2 s.

Figures

Figures reproduced from arXiv: 1908.10235 by the authors.

Figure 1
Figure 1. Block diagram of the proposed system. The initial inputs of the system are fixed and moving images down-scaled by a factor of four ( 4). Three RegNets process the input images over three stages (4, 2, 1) and generate the final output T (x) = Ts1 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. The generation of training pairs from a single input image IM0. The input image is deformed slightly using the single frequency category, with the “lowest” settings (see Table I), to generate moving images IM i. These are then each deformed and post-processed multiple times using all categories to generate fixed images IF i. C. Artificial generation of DVFs and images In order to train a CNN, a considerable number o… view at source ↗
Figure 2
Figure 2. RegNet designs: The inputs of the U-Net design are entire down-scaled images. However, in the Multi-view and U-Net-advanced architectures the output size is smaller than the input size and can be trained in a patch-based manner. correspondence of all three pipelines. Finally, all pipelines are merged together followed by three more convolutional layers. The network gives three 3D outputs of size 21 × 21 × 21 corresp… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Examples of heat maps of generated artificial DVFs overlayed on the deformed images. We show three of the five spatial frequencies defined in Table I. The histogram of the Jacobian determinant of each DVF is shown next to the sample image. As the spatial frequency incr…
Figure 5
Figure 5. Figure 5: Example results (top row) and difference images (bottom row) from DIR-Lab-4DCT study. TABLE III: Quantitative results of the SPREAD study in the training set (case 1 to case 11) and in the test set (case 13 to case 21). This experiment is performed with the network com…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TotalRegistrator: Towards a Lightweight Foundation Model for CT Image Registration

    eess.IV 2025-08 conditional novelty 6.0 of 10

    A single-center whole-body CT registration model with region-specific deformation blocks matches or beats a foundation model on abdominal organs and generalizes to external datasets without fine-tuning.

  2. CardioMorphNet: Cardiac Motion Prediction Using a Shape-Guided Bayesian Recurrent Deep Network

    cs.CV 2025-08 conditional novelty 5.0 of 10

    CardioMorphNet estimates cardiac motion by learning to warp predicted heart-shape masks through a recurrent Bayesian network, reporting top results on UK Biobank cine MRI.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages · cited by 2 Pith papers

  1. [6]

    K. A. Eppenhof, M. W. Lafarge, P. Moeskops, M. Veta, J. P. Pluim, Deformable image registration using convolutional neural networks, in: Medical Imaging 2018: Image Pro- cessing, vol. 10574, International Society for Optics and Photonics, 105740S,

    work page 2018

  2. [7]

    Y . Hu, E. Gibson, N. Ghavami, E. Bonmati, C. M. Moore, M. Emberton, T. Vercauteren, J. A. Noble, D. C. Bar- ratt, Adversarial Deformation Regularization for Train- ing Image Registration Neural Networks, arXiv preprint arXiv:1805.10665 . K. Ma, J. Wang, V . Singh, B. Tamersoy, Y .-J. Chang, A. Wim- mer, T. Chen, Multimodal Image Registration with Deep Co...

    work page Pith review arXiv

  3. [12]

    K. A. Eppenhof, J. P. Pluim, Supervised local error estimation for nonlinear image registration using convolutional neural networks, in: SPIE Medical Imaging, International Society for Optics and Photonics, 101331U–101331U, 2017

    work page 2017

  4. [78]

    M. Hub, M. L. Kessler, C. P. Karger, A stochastic approach to estimate the uncertainty involved in B-spline image registration, IEEE Transactions on Medical Imaging 28 (11) (2009) 1708–1716. M. Staring, M. Bakker, J. Stolk, D. Shamonin, J. Reiber, B. Stoel, Towards local progression estimation of pulmonary emphysema using CT, Medical Physics 41 (2) (2014)...

    work page 2009

  5. [170]

    S. E. Muenzing, B. van Ginneken, K. Murphy, J. P. Pluim, Su- pervised quality assessment of medical image registration: Application to intra-patient CT lung registration, Medical image analysis 16 (8) (2012) 1521–1531. H. Sokooti, G. Saygili, B. Glocker, B. P. Lelieveldt, M. Staring, Accuracy Estimation for Medical Image Registration Using Regression Fore...

    work page 2012

  6. [1849]

    Castillo, E

    R. Castillo, E. Castillo, D. Fuentes, M. Ahmad, A. M. Wood, M. S. Ludwig, T. Guerrero, A reference dataset for deformable image registration spatial accuracy evaluation using the COPDgene study archive, Physics in Medicine & Biology 58 (9) (2013)

    work page 2013

  7. [2008]

    S. Hu, L. Wei, Y . Gao, Y . Guo, G. Wu, D. Shen, Learning- based deformable image registration for infant MR images in the first year of life, Medical Physics 44 (1) (2017) 158–

    work page 2017

  8. [2015]

    Kamnitsas, C

    K. Kamnitsas, C. Ledig, V . F. Newcombe, J. P. Simpson, A. D. Kane, D. K. Menon, D. Rueckert, B. Glocker, Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation, Med. Image Anal. 36 (2017) 61–

    work page 2017

Show all 12 references
  1. [2016]

    Sokooti, G

    H. Sokooti, G. Saygili, B. Glocker, B. P. Lelieveldt, M. Staring, Quantitative Error Prediction of Medical Image Registration using Regression Forests, Med. Image Anal. . S. Miao, Z. J. Wang, R. Liao, A CNN regression approach for real-time 2D/3D registration, IEEE Transaction...

    2016

  2. [2017]

    B. D. de V os, F. F. Berendsen, M. A. Viergever, H. Sokooti, M. Staring, I. I ˇsgum, A deep learning framework for un- supervised affine and deformable image registration, Med. Image Anal. 52 (2019) 128–143. G. Balakrishnan, A. Zhao, M. R. Sabuncu, J. Guttag, A. V . Dalca, An U...

    2019 arXiv

  3. [2018]

    M. S. Elmahdy, J. M. Wolterink, H. Sokooti, I. Iˇsgum, M. Star- ing, Adversarial optimization for joint registration and segmentation in prostate CT radiotherapy, arXiv preprint arXiv:1906.12223 . A. Sheikhjafari, M. Noga, K. Punithakumar, N. Ray, Unsu- pervised deformable ima...

    1906 arXiv

  4. [2861]

    Murphy, B

    K. Murphy, B. van Ginneken, S. Klein, M. Staring, B. J. de Hoop, M. A. Viergever, J. P. Pluim, Semi-automatic construction of reference standards for evaluation of image registration, Med. Image Anal. 15 (1) (2011) 71–84. F. F. Berendsen, A. N. Kotte, M. A. Viergever, J. P. Pl...

    2011

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.