Pith. sign in

REVIEW 4 major objections 7 minor 45 references

Collaborative Learning for Unsupervised Multimodal Remote Sensing Image Registration: Integrating Self-Supervision and MIM-Guided Diffusion-Based Image Translation

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Three networks that train each other can register multimodal images with no geometric labels, matching or beating supervised baselines.

desk verdict Novel collaborative pseudo-label framework for unsupervised multimodal registration with promising empirical results, but the teacher loop is unvalidated and the manuscript has several fixable mechanical errors. read the letter →

arxiv 2505.22000 v1 pith:CY2JIJ5T submitted 2025-05-28 eess.IV

Xiaochen Wei , Weiwei Guo , Wenxian Yu This is my paper
classification eess.IV
keywords multimodalimageregistrationunsupervisedlearningconditionaldiffusionmodelMaximumIndexMapimage-to-imagetranslationself-supervisedpseudo-labelshomographyestimationremotesensing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CoLReg claims that accurate registration of images from different sensors—optical, infrared, SAR, depth, and low-light visible—can be learned without any ground-truth geometric labels, by having three networks train each other. A diffusion-based image translator synthesizes cross-modal pairs with known simulated deformations, a self-supervised teacher registration network learns on those pairs, and a student network is trained on real cross-modal pairs using the teacher's outputs as pseudo-labels; the student's own estimates then help refine the translator. The paper reports that this loop reaches or beats state-of-the-art unsupervised methods on five datasets and outperforms several fully supervised baselines, including on hard modality gaps. If this holds, the main bottleneck of supervised registration—expensive manual annotation of control points—disappears, and unsupervised training could scale to arbitrary sensor combinations.

What carries the argument

The load-bearing machinery has two parts. The first is the Maximum Index Map (MIM): a handcrafted, modality-invariant feature computed by summing wavelet response magnitudes across scales at each pixel and taking the maximum across orientations, which highlights contours and structures that persist across sensors; MIMGCD conditions its diffusion reverse process on MIM-like features so that translated images preserve geometry. The second is the pseudo-label loop: the self-supervised intermediate network $R_C^s$ is trained on synthetic pairs $\{ \boldsymbol{x}_{S'}, \boldsymbol{x}_{S\to T} \}$ with exact simulated displacement labels, and Eq. 18 then lets it generate pseudo-labels $\Delta P_{PL} = R_C^s(\boldsymbol{x}_S, \boldsymbol{x}_T)$ that supervise the distilled network $R_C$ on real pairs. An alternating schedule (Algorithm 1) updates translator, teacher, and student in sequence, and the student's estimated homography warps $\boldsymbol{x}_T$ into $\boldsymbol{x}_{T,W}$ to supervise the translator—the mechanism by which the paper claims the three networks mutually reinforce.

What would settle it

Hold out a set of real cross-modal pairs with manually annotated corner correspondences, train CoLReg without touching them, and plot both the intermediate network's pseudo-label error and the final network's corner error after each alternating round. If pseudo-label error stays flat or grows while final accuracy improves, the collaborative loop is not the source of the gains; if freezing the teacher after the first round reproduces the full loop's accuracy, the alternating optimization is not load-bearing. The same data would settle the paper's 'surpasses supervised' claim, since the supervised baselines here are trained on synthetically deformed pairs from the same five datasets rather than on held-out real geometry.

Watch

Extended reading notes

Core claim

The paper's central claim is that direct cross-modal registration can be learned without any ground-truth geometric labels by replacing the standard 'translate, then register in one modality' pipeline with an alternating three-way self-training loop. The translation network (MIMGCD) is a conditional diffusion model conditioned on a Maximum Index Map, a modality-invariant wavelet feature that emphasizes boundaries and structure, so the cross-modal pairs it synthesizes retain the geometric cues registration needs. The intermediate registration network $R_C^s$ trains on those synthetic pairs together with known simulated transformations $\tilde{H}_{GT}$, then produces pseudo-labels $\Delta P_{PL} = R_C^s(\boldsymbol{x}_S, \boldsymbol{x}_T)$ for real cross-modal pairs, and the final network $R_C$ is trained against those pseudo-labels. The student's estimated transformation warps the target image, and the warped image becomes supervision for the translator, closing the loop. The paper reports that this loop reaches accuracy competitive with or better than several supervised baselines on five multimodal datasets, and argues that training the student directly on real cross-modal pairs makes it learn modality-invariant geometric features instead of inheriting translation errors.

Load-bearing premise

The final network is supervised only by pseudo-labels from the intermediate network, and nothing in the training loop ever checks those pseudo-labels against true geometric correspondences—if the teacher is systematically wrong, the student inherits and amplifies the bias.

Editorial extensions

If this is right

  • Multimodal registration pipelines can drop the expensive step of annotating control points: the same three-network loop provides the geometric supervision that supervised methods obtain from ground-truth homographies.
  • Because the student is trained directly on real cross-modal pairs, registration accuracy is no longer bounded by translation quality—a limit the paper identifies in translate-then-register pipelines such as SSHNet.
  • The pseudo-labels should improve as the loop alternates, so the framework is expected to keep gaining accuracy with more alternating iterations rather than plateauing at the teacher's initial level.
  • On hard modality gaps (depth-visual, SAR-optical, low-light visible-infrared) where unsupervised baselines collapse, the paper reports CoLReg retains meaningful registration accuracy, suggesting the MIM guidance preserves geometric cues those methods lose.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The loop is a self-training scheme whose correctness is never checked: the paper does not measure $R_C^s$'s pseudo-labels against held-out geometric correspondences, so a direct test is to track pseudo-label error across alternating rounds on annotated hold-out pairs and see whether it shrinks or compounds.
  • Because the translator, the teacher, and the student share the loop, the reported gains could come from the MIM guidance, the alternating schedule, or the synthetic-pair pretraining alone; an ablation that freezes each component in turn (the paper's $IT=1$ variant partially does this) would attribute the improvement.
  • MIM is a fixed handcrafted conditioner; a natural extension is to learn the conditioner end-to-end and compare, which would show whether geometric preservation is best enforced by explicit wavelet guidance or by the registration loss itself.
  • The 'beats supervised baselines' claim is measured on synthetically deformed test pairs drawn from the same datasets used in training; testing on genuinely unseen sensors or acquisition geometries would be the sharper comparison, and the zero-shot results in the paper already suggest where that advantage may erode.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes CoLReg, an unsupervised multimodal remote-sensing image registration framework that combines three networks in an alternating optimization loop: a MIM-guided conditional diffusion model (MIMGCD) for cross-modal image translation, an intermediate self-supervised registration network SRegNet trained on synthetic cross-modal pairs with known homographies, and a distilled cross-modal registration network URegNet trained on real unaligned pairs using pseudo-labels produced by SRegNet. The authors report results on five multimodal datasets, claiming competitive or superior performance against state-of-the-art unsupervised methods and several supervised baselines, with additional ablations and a zero-shot generalization study. The central mechanism is a pseudo-label self-training loop whose soundness depends on the accuracy of SRegNet's predictions on real cross-modal pairs and on geometric preservation by the diffusion-based translator.

Significance. If the framework works as described, it is a notable advance: it would demonstrate that a fully unsupervised pipeline, without any ground-truth geometric supervision on real pairs, can compete with or beat supervised deep homography networks on hard multimodal remote-sensing data. The paper has clear strengths: a wide experimental comparison across five datasets, ablations that isolate the contribution of the alternating optimization and of the MIMGCD translator, and a zero-shot generalization study. However, the significance is currently tempered by two kinds of unresolved issues: (i) load-bearing technical details that prevent independent reimplementation (the self-referential MIM definition in Eq. 7 and the incorrect inference call in Algorithm 2), and (ii) the absence of any direct measurement of pseudo-label quality, which is the linchpin of the claimed unsupervised advantage.

major comments (4)
  1. [§3.2.3, Eq. (18); Algorithm 1] The final network R_C is trained exclusively on pseudo-labels ΔP_PL = R_C^s(x_S, x_T) produced by the intermediate network on real cross-modal pairs, and Algorithm 1 initializes R_C with R_C^s's parameters (line 5) before retraining on those pseudo-labels (line 17). Because the training pairs were generated by applying a known random homography to aligned pairs (as stated in §4.1.1), the authors can directly evaluate the pseudo-label accuracy of R_C^s against the true homography on a held-out portion of the real training pairs, but no such measurement is reported. If the teacher has a systematic bias, the student inherits and possibly amplifies it, and the loop contains no real-pair geometric ground truth to catch this. Please add a quantitative assessment of pseudo-label ACE on real training pairs, and ideally an ablation that removes the pseudo-label fine-tuning step to show its marginal contribution.
  2. [§3.2.1, Eq. (7)] The definition of the Maximum Index Map is self-referential: E_so(x,y) and O_so(x,y) on the right-hand side are defined as filter responses of x_T_MIM(x,y), which is itself the quantity being defined in the same equation. As written, Eq. (7) is not computable, and since MIMGCD and all downstream synthetic-pair training depend on this feature, the method cannot be re-implemented from the manuscript. Please restate the equation with the input image (e.g., x_T) in the filter responses rather than x_T_MIM.
  3. [Algorithm 2 (inference)] The inference algorithm is inconsistent with the rest of the method. Its input is the pretrained network R_C(Φ^(IT-1)_c), but the only call in the algorithm is Δ̂P^ST_Ni = R_C^s(x_S, x_T, Φ^(it)_c), which invokes the intermediate network and uses an undefined parameter symbol `it` after training has ended. As written, the reported test results cannot be reproduced because the inference procedure does not actually use the trained R_C. Please correct this to a call of R_C with the final parameters Φ^(IT-1)_c.
  4. [§3.2.2, Eq. (17)] The self-supervised training of R_C^s uses pairs {x_S', x_S→T} with displacement labels ΔP_GT^s obtained by parameterizing the simulated homography H_GT^s between x_S' and x_S. These labels are correct only if the translated image x_S→T is geometrically identical to x_S; however, x_S→T is generated by the stochastic reverse process of a conditional diffusion model (Eq. 14), and the paper never verifies that MIMGCD preserves the geometry of the source image. This is a load-bearing assumption for the teacher's training signal, and it should be checked directly, e.g., by computing ACE between x_S→T and x_S on the synthetic pairs or by measuring the repeatability of known control points before and after translation.
minor comments (7)
  1. [Table 2] In the GoogleEarth block, the value for 'Our' under AUC@10 is listed as 7.80, which is inconsistent with the same configuration reported as 87.80 in Table 4; this appears to be a typographical error that should be corrected.
  2. [Throughout] The method name is inconsistent across the manuscript: CoLReg, MGCD-MUReg, MGC-UMReg, and CoDiMIR are all used in different places; please unify the name in the text, tables, and figure captions.
  3. [Algorithm 1, lines 8 and 12] The variable names in the loading steps appear swapped: line 8 loads Ψ_T^MIM with Φ_s^(it-1)_mim, and line 12 loads Ψ_S^MIM with Φ_t^(it)_mim, which contradicts the loss definitions in Eqs. (12) and (14)-(16); please correct these parameter references.
  4. [§4.4] The zero-shot analysis refers to 'Fig. ??' and the accompanying figure seems to be missing; without the figure or a table of zero-shot numbers, the claims in this section cannot be verified.
  5. [§4.1.3 / Table 2 caption] The text states that inference time and memory consumption are reported, but Table 2 only includes inference time; either add the memory column or adjust the claim in the caption.
  6. [Eq. (14)] There is a typo in the subscript of the diffusion parameters: 'Φ^(it)_dff' should be 'Φ^(it)_diff'.
  7. [§5 Conclusion] The phrase 'treecomponents' should read 'three components'.

Circularity Check

3 steps flagged · score 4.0 of 10

Pseudo-label self-distillation, a self-referential MIM definition, and a closed translation-registration loop make the real-data training path partially circular; external test benchmarks keep the central claim independently testable.

  1. fitted input called prediction [Section 3.2.3, Eq. (18); Algorithm 1 lines 5 and 16]
    "As shown in Fig.5, 𝑅𝐶 has the same architecture with 𝑅𝐶𝑠. ... the loss for training 𝑅𝐶 is given by: ℒ𝑈 = ℒ𝑑𝑖𝑠𝑝({Δ̂𝑃𝑖}𝑖=𝑁𝑖𝑖=1 ,Δ𝑃𝑃); {Δ̂𝑃𝑖}𝑖=𝑁𝑖𝑖=1 = 𝑅𝐶(𝑥𝑆, 𝑥𝑇,𝛷𝑐); Δ𝑃𝑃 = Δ̂𝑃𝑆𝑇𝑁𝑖; {Δ̂𝑃𝑆𝑇𝑖}𝑖=𝑁𝑖𝑖=1 = 𝑅𝐶𝑠(𝑥𝑆, 𝑥𝑇,𝛷𝑠)."

    The real-pair supervision for the final network is, by Eq. (18), the output of the intermediate network on the same input pair: Δ𝑃𝑃 = 𝑅𝐶𝑠(𝑥𝑆, 𝑥𝑇). Algorithm 1 initializes 𝑅𝐶 with 𝑅𝐶𝑠's parameters (lines 5 and 16), and the paper states the two networks share the same architecture. Training 𝑅𝐶 on the real pairs is therefore a distillation that by construction minimizes its output difference from the teacher; it injects no geometric information beyond what 𝑅𝐶𝑠 already predicts. The claimed 'mutual collaboration ... enhances the quality of pseudo-labels' is a self-consistency loop, since no real-pair ground-truth correspondence enters Eq. (18) to detect or correct systematic teacher bias.

  2. self definitional [Section 3.2.1, Eq. (7)]
    "𝒙𝑇MIM(𝑥,𝑦)=max( 𝐴𝑜(𝑥,𝑦)𝑁𝑜1 ) ... 𝐸𝑠𝑜(𝑥,𝑦)= 𝒙𝑇MIM(𝑥,𝑦)∗ 𝐿𝑒𝑣𝑒𝑛(𝑥,𝑦,𝑠,𝑜 ); 𝑂𝑠𝑜(𝑥,𝑦)= 𝒙𝑇MIM(𝑥,𝑦)∗ 𝐿𝑜𝑑𝑑(𝑥,𝑦,𝑠,𝑜 )."

    The Maximum Index Map that conditions the diffusion model is defined recursively: the even and odd wavelet responses 𝐸𝑠𝑜 and 𝑂𝑠𝑜 are computed by convolving 𝒙𝑇MIM itself with wavelets, and 𝒙𝑇MIM is then the max over orientations of the resulting magnitude. As printed, the feature is a fixed point of its own definition rather than a function of the input image 𝒙𝑇, so the conditioning signal used throughout MIMGCD (Eqs. (5)-(14)) is not grounded in the image content. This is self-definitional: the 'modality-invariant prior' underpinning the whole translation mechanism is defined in terms of itself, making the pipeline non-reproducible as written.

1 more flagged steps
  1. self definitional [Section 3.1 Eq. (4); Section 3.2.1 Eq. (14) and Fig. 4(E)]
    "Due to the lack of well aligned image pairs(𝑥𝑆, 𝑥𝑇) in the unsupervised setting, we utlize the estimated 𝐻 to wrap the 𝑥𝑇, yielding the cross-modal image pairs{(𝑥𝑆, 𝑥𝑇,𝑊)} to train the image translation network ... the 𝑥𝑇 is wrapped by an estimated transformation by the trained registration network to guide the training of the image-to-image translation network, thereby forming a closed-loop collaborative learning cycle."

    MIMGCD's translation supervision at 𝑖𝑡≥1 is 𝑥𝑇,𝑊 = warp(𝑥𝑇, 𝐻̂−1), where 𝐻̂ is estimated by the final registration network (Eq. (14), Fig. 4(E)). That final network is trained, in Eq. (18), on pseudo-labels produced by the intermediate network, while the intermediate network is trained on pairs 𝑥𝑆→𝑇 generated by MIMGCD (Eq. (3)). Thus the translation network, the intermediate network, and the final network are each defined through the others' outputs on the real-data branch, with no independent real-pair geometric correspondence entering the cycle. The 'closed-loop collaborative learning cycle' can only enforce internal consistency among the three networks, not accuracy against true real-pair geometry; any claim that the cycle improves pseudo-label quality is self-referential.

full rationale

The central empirical claim is not fully circular: the final network is evaluated on held-out test pairs with ground-truth homographies, and the intermediate network's synthetic branch (Eqs. (3) and (17)) uses real ground-truth transformations 𝐻𝐺𝑇, so independent evidence can in principle support the benchmark results. However, the real-data refinement path is a self-training/distillation loop: Eq. (18) makes the student's training target the teacher's own output on the same real pair, Eq. (7) defines the MIM feature recursively in terms of itself, and Eqs. (4)/(14) define translation supervision through the registration estimate. These are concrete self-referential reductions in the training derivation, not merely missing ablations. The only self-citation (Wei et al. 2025 for the one-step reverse process) inherits a design choice but is not load-bearing for the comparison, so it does not by itself raise the score. Overall the real-data component is partially circular, while the synthetic anchor and external evaluation keep the central claim independently testable; score 4.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The central result rests on a large number of hand-chosen training hyperparameters, several unstated domain assumptions about the invariance of MIM features and the reliability of pseudo-labels, and the adopted MCNet backbone. No new physical entities are introduced.

free parameters (8)
  • Number of alternating optimization rounds IT = not specified
    Controls how many times the three networks reinforce each other; no value or stopping rule is given in Sec. 3.2.4 or Algorithm 1.
  • Loss weight lambda_mds in Eq. 16 = not specified
    Weights the translated-image consistency loss in the source MIM encoder objective; the paper does not report its value or a sensitivity study.
  • Diffusion noise schedule beta_t and timesteps t1, t2 = not specified
    The schedule controls the forward/reverse diffusion in Eqs. 5 and 6, but the actual schedule and timestep values used in training are omitted.
  • MIM scales N_s and orientations N_o = not specified
    The MIM computation in Eq. 7 depends on these counts; no values are given.
  • Training iteration counts and learning rates per stage = 600K, 300K, 30K, 60K, 10K iterations; LR 2.5e-4 and 4e-4
    Hand-chosen training schedule in Sec. 4.1.4; the central results depend on these choices.
  • Random perspective transformation range for synthetic H_GT^s = not specified
    The self-supervised intermediate network is trained with randomly warped source images in Eq. 3; the deformation magnitude is never stated.
  • Registration network iterations q_sc per scale and total N_i = not specified; MCNet uses four scales
    MCNet's scale-wise iteration counts and total iterations are not reported for the adapted backbone.
  • Batch size = 16
    Set in Sec. 4.1.4; not varied across datasets.
assumptions (6)
  • standard math Diffusion forward process and Tweedie's formula are valid for computing clean samples in one step.
    Used in Eqs. 5, 8, and 9 to add noise and to recover the translated image in one reverse step.
  • domain assumption The Maximum Index Map is modality-invariant and preserves geometric structure.
    Invoked in Sec. 3.2.1 to justify using MIM/MIM-like features as diffusion conditioning; no quantitative support is given in this paper.
  • ad hoc to paper The estimated homography H from the registration network is accurate enough to warp the target image into a useful translation supervision signal.
    In Eq. 4 and Fig. 4(E), x_T_W = warp(H, x_T) is used as supervision for MIMGCD; this assumes the current registration estimate is already reliable.
  • ad hoc to paper Pseudo-labels from the intermediate network on real cross-modal pairs are reliable enough to supervise the final registration network.
    Eq. 18 uses Delta_P_PL from R_C^s as the only supervision for R_C on real pairs.
  • domain assumption MCNet is a strong fixed registration backbone and its architectural choices transfer to the new training paradigm.
    The paper adopts MCNet with four scales in Sec. 3.2.2 without re-evaluating this choice.
  • domain assumption The multimodal datasets contain homography-induced geometric deformation and the synthetic perturbation model covers the test distribution.
    Test pairs are generated by applying random homographies to aligned pairs using the DeTone et al. (2016) perturbation method; natural misalignment may not follow this model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Collaborative Learning for Unsupervised Multimodal Remote Sensing Image Registration: Integrating Self-Supervision and MIM-Guided Diffusion-Based Image Translation." pith.science (2026). https://pith.science/paper/CY2JIJ5T

@misc{pith2026250522000,
  author       = {Pith},
  title        = {Pith review of: Collaborative Learning for Unsupervised Multimodal Remote Sensing Image Registration: Integrating Self-Supervision and MIM-Guided Diffusion-Based Image Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CY2JIJ5T}},
  note         = {Machine review of arXiv:2505.22000}
}
read the original abstract

The substantial modality-induced variations in radiometric, texture, and structural characteristics pose significant challenges for the accurate registration of multimodal images. While supervised deep learning methods have demonstrated strong performance, they often rely on large-scale annotated datasets, limiting their practical application. Traditional unsupervised methods usually optimize registration by minimizing differences in feature representations, yet often fail to robustly capture geometric discrepancies, particularly under substantial spatial and radiometric variations, thus hindering convergence stability. To address these challenges, we propose a Collaborative Learning framework for Unsupervised Multimodal Image Registration, named CoLReg, which reformulates unsupervised registration learning into a collaborative training paradigm comprising three components: (1) a cross-modal image translation network, MIMGCD, which employs a learnable Maximum Index Map (MIM) guided conditional diffusion model to synthesize modality-consistent image pairs; (2) a self-supervised intermediate registration network which learns to estimate geometric transformations using accurate displacement labels derived from MIMGCD outputs; (3) a distilled cross-modal registration network trained with pseudo-label predicted by the intermediate network. The three networks are jointly optimized through an alternating training strategy wherein each network enhances the performance of the others. This mutual collaboration progressively reduces modality discrepancies, enhances the quality of pseudo-labels, and improves registration accuracy. Extensive experimental results on multiple datasets demonstrate that our ColReg achieves competitive or superior performance compared to state-of-the-art unsupervised approaches and even surpasses several supervised baselines.

Figures

Figures reproduced from arXiv: 2505.22000 by the authors.

Figure 1
Figure 1. Unsupervised multimodal image registration methods in early research. (a) registration methods based on cross￾modality measure; (b) registration methods based on image-to-image translation; (c) split registration learning with image-to￾image translation and mono-modal registration; (d) our direct alternative registration learning method based on image-to-image translation and cross-modality registration learning. bo… view at source ↗
Figure 2
Figure 2. Overview of alternating optimization in CoLReg (Collaborative Learning for Unsupervised Multimodal Image Registration). (a) Optimize MIMGCD  (𝛷𝑡 ); (b) Optimize SRegNet 𝐶 𝑠 (𝛷𝑠 ); (c) Optimize URegNet 𝐶 (𝛷𝑐 ). Since the training involves multi-objective losses 𝑐 ,𝑠 ,𝑡 optimization, we adopt an alternating optimization scheme to ensure stable convergence and mutual reinforcement. The proposed framework is illus… view at source ↗
Figure 4
Figure 4. The detail optimization of MIMGCD  = {Ψ𝑡 MIM, Ψ𝑠 MIM, Ψdif f }(𝑖𝑡 >= 1). (A): The training of Ψ𝑡 MIM(𝛷 𝑡(𝑖𝑡) mim); (B): The training of Ψ𝑠 MIM(𝛷 𝑠(𝑖𝑡) mim); (C): The training of Ψ 𝑠(𝑖𝑡) MIM; (D): The network architecture of Ψdif f (𝛷 (𝑖𝑡) dif f ); (E): The obtaining of pseudo￾labels 𝒙 𝑇 ,𝑊 and predicted transformations 𝐻̂ by the 𝐶 (𝛷(𝑖𝑡−1) 𝑐 ). to mask the padding pixels in 𝒙 𝑇 ,𝑊 . Since 𝒙 𝑇 ,𝑊 is not the exact g… view at source ↗
Figures from the paper (7 more)
Figure 7
Figure 7. Figure 7: The traing of cross-modality registration network 𝐶 (𝛷(𝑖𝑡) 𝑐 ) with unsupervised cross-modality image pair 𝑥 𝑆 , 𝑥𝑇 . Δ𝑃 𝑃 𝑙 is the pseudo-label for supervising the predicted displace￾ments of four corners {Δ𝑃̂ 𝑖 } 𝑖=𝑁𝑖 𝑖=1 Algorithm 1 The training for CoLReg Require:…
Figure 5
Figure 5. Figure 5: The network architecture of Intermediate Regis￾tration Network 𝐶 𝑠 and registration network 𝐶 , which is MCNetZhu et al. (2024) with four scales. The 𝒙1 and 𝒙2 are input images, which are firstly input into shared feature encoder 𝐶 𝑠𝑓 or 𝐶 𝑓 to extract multiscale f…
Figure 6
Figure 6. Figure 6: The training of Intermediate Registration Network 𝐶 𝑠 with cross-modality image pair {𝒙 𝑆 ′ , 𝒙 𝑆→𝑇 }. Δ𝑃 𝐺𝑇 𝑠 is the displacements of four corners by parameterizing the matrix 𝐻𝐺𝑇 𝑠 . {Δ𝑃̂ 𝑆 ′𝑆 𝑖 } 𝑖=𝑁𝑖 𝑖=1 is the predicted displacements of corners, and 𝑁𝑖 is the num…
Figure 8
Figure 8. Figure 8: Qualitative image registration results of our method MGCD-UReg, supervised methods: DHN, MHM, ReDFeat, IHN1, and unsupervised methods: SCPNet, Alto, SSHNet, SSHNet-D on easy dataset GoogleEarth and RGB_IR_AI. Green polygons denote the ground-truth deformation from the …
Figure 9
Figure 9. Figure 9: Qualitative image registration results of our method MGCD-UReg, supervised methods: DHN, MHM, IHN1, and unsupervised methods: SCPNet, Alto, SSHNet, SSHNet-D on the challenge dataset. Green polygons denote the ground-truth deformation from source images to target images…
Figure 10
Figure 10. Figure 10: (a): CoLReg trained with self-supervised data and without alternative optimization(IT == 1); (b): CoLReg trained with unsupervised data and IT == 1; (c) CoLReg trained with unsupervised data and with alternative optimization [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Compare the zero-shot image registration performance of different methods using "GoogleEarth+RGB_IR_AI" as an example, where GoogleEarth serves as the training set and RGB_IR_AI as the test set. performance by testing it on the test sets of the remaining unseen datase…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 36 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in ":" * " " * FUNCTION f...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in ":" * " " * FUNCTION f...

  3. [3]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in ":" * " " * FUNCTION f...

  4. [4]

    , author Ginger, Y

    author Arar, M. , author Ginger, Y. , author Danon, D. , author Bermano, A.H. , author Cohen-Or, D. , year 2020 . title Unsupervised multi-modal image registration via geometry preserving image-to-image translation , in: booktitle Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. pages 13410--13419

    work page 2020

  5. [5]

    , author Hu, J

    author Cao, S.Y. , author Hu, J. , author Sheng, Z. , author Shen, H.L. , year 2022 . title Iterative deep homography estimation , in: booktitle Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. pages 1879--1888

    work page 2022

  6. [6]

    , author Zhang, R

    author Cao, S.Y. , author Zhang, R. , author Luo, L. , author Yu, B. , author Sheng, Z. , author Li, J. , author Shen, H.L. , year 2023 . title Recurrent homography estimation using homography-guided image warping and focus transformer , in: booktitle Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. pages 9833--9842

    work page 2023

  7. [7]

    , author Wei, J

    author Chen, Z. , author Wei, J. , author Li, R. , year 2022 . title Unsupervised multi-modal medical image registration via discriminator-free image-to-image translation . journal arXiv preprint arXiv:2204.13656

    arXiv 2022

  8. [8]

    , author Min, D

    author Cho, J. , author Min, D. , author Kim, Y. , author Sohn, K. , year 2021 . title Diml/cvl rgb-d dataset: 2m rgb-d images of natural indoor and outdoor scenes . journal arXiv preprint arXiv:2110.11590

    arXiv 2021

Show all 45 references
  1. [9]

    , author Ma, A

    author Cui, S. , author Ma, A. , author Wan, Y. , author Zhong, Y. , author Luo, B. , author Xu, M. , year 2022 . title Cross-modality image matching network with modality-invariant feature representation for airborne-ground thermal infrared and visible datasets . journal IEEE...

    2022

  2. [10]

    , author Luqman, M.M

    author Dang, Q.B. , author Luqman, M.M. , author Coustaty, M. , author Tran, D.C. , author Ogier, J.M. , year 2015 . title Srif: Scale and rotation invariant features for camera-based document image retrieval . journal 2015 13th International Conference on Document Analysis an...

    2015

  3. [11]

    , author Ma, J

    author Deng, Y.C. , author Ma, J. , year 2022 . title Redfeat: Recoupling detection and description for multimodal feature learning . journal IEEE Transactions on Image Processing volume 32 , pages 591--602 . https://api.semanticscholar.org/CorpusID:248811367

    2022

  4. [12]

    , author Malisiewicz, T

    author DeTone, D. , author Malisiewicz, T. , author Rabinovich, A. , year 2016 . title Deep image homography estimation . journal arXiv preprint arXiv:1606.03798

    2016 arXiv

  5. [13]

    , author Malisiewicz, T

    author DeTone, D. , author Malisiewicz, T. , author Rabinovich, A. , year 2018 . title Superpoint: Self-supervised interest point detection and description , in: booktitle Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pp. pages 224--236

    2018

  6. [14]

    , year 2025

    author Guo, M. , year 2025 . title Unsupervised multi-modal medical image registration via invertible translation , in: booktitle European Conference on Computer Vision , organization Springer . pp. pages 22--38

    2025

  7. [15]

    , author Liu, M.Y

    author Huang, X. , author Liu, M.Y. , author Belongie, S.J. , author Kautz, J. , year 2018 . title Multimodal unsupervised image-to-image translation , in: booktitle European Conference on Computer Vision . https://api.semanticscholar.org/CorpusID:4883312

    2018

  8. [16]

    , author Zhu, C

    author Jia, X. , author Zhu, C. , author Li, M. , author Tang, W. , author Zhou, W. , year 2021 . title Llvip: A visible-infrared paired dataset for low-light vision , in: booktitle Proceedings of the IEEE/CVF international conference on computer vision , pp. pages 3496--3504

    2021

  9. [17]

    , author Kwon, G

    author Kim, B. , author Kwon, G. , author Kim, K. , author Ye, J.C. , year 2024 . title Unpaired image-to-image translation via neural schrödinger bridge , in: booktitle ICLR

    2024

  10. [18]

    , author Kim, M

    author Kim, J. , author Kim, M. , author Kang, H. , author Lee, K.H. , year 2020 . title U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation , in: booktitle International Conference on Learning Repres...

    2020

  11. [19]

    , author Arani, E

    author Koguciuk, D. , author Arani, E. , author Zonooz, B. , year 2021 . title Perceptual loss for robust unsupervised homography estimation . journal 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 4269--4278 https://api.semantics...

    2021

  12. [20]

    , author Qi, X.S

    author Kong, L. , author Qi, X.S. , author Shen, Q. , author Wang, J. , author Zhang, J. , author Hu, Y. , author Zhou, Q. , year 2023 . title Indescribable multi-modal spatial evaluator , in: booktitle Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2023

  13. [21]

    , author Liu, F

    author Le, H. , author Liu, F. , author Zhang, S. , author Agarwala, A. , year 2020 . title Deep homography estimation for dynamic scenes , in: booktitle Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. pages 7652--7661

    2020

  14. [22]

    , author Hu, Q

    author Li, J. , author Hu, Q. , author Ai, M. , year 2019 . title Rift: Multi-modal image matching based on radiation-variation insensitive feature transform . journal IEEE Transactions on Image Processing volume 29 , pages 3296--3310 . https://api.semanticscholar.org/CorpusID...

    2019

  15. [23]

    , author Shao, H.C

    author Li, Y. , author Shao, H.C. , author Qian, X. , author Zhang, Y. , . title Fddm: Unsupervised mr-to-ct translation with a frequency-decoupled diffusion model

  16. [24]

    , author He, W

    author Liu, Y. , author He, W. , author Zhang, H. , year 2024 . title Grid: Guided refinement for detector-free multimodal image matching . journal IEEE Transactions on Image Processing volume 33 , pages 5892--5906 . https://api.semanticscholar.org/CorpusID:273288273

    2024

  17. [25]

    , author Huang, H

    author Lv, Z. , author Huang, H. , author Sun, W. , author Lei, T. , author Benediktsson, J.A. , author Li, J. , year 2023 . title Novel enhanced unet for change detection using multimodal remote sensing image . journal IEEE Geoscience and Remote Sensing Letters volume 20 , pa...

    2023

  18. [26]

    , author Efros, A.A

    author Park, T. , author Efros, A.A. , author Zhang, R. , author Zhu, J.Y. , year 2020 . title Contrastive learning for unpaired image-to-image translation , in: booktitle European Conference on Computer Vision

    2020

  19. [27]

    , author Wang, S

    author Quan, D. , author Wang, S. , author Gu, Y. , author Lei, R. , author Yang, B. , author Wei, S. , author Hou, B. , author Jiao, L. , year 2022 . title Deep feature correlation learning for multi-modal remote sensing image registration . journal IEEE Transactions on Geosc...

    2022

  20. [28]

    , author Jurie, F

    author Razakarivony, S. , author Jurie, F. , year 2016 . title Vehicle detection in aerial imagery : A small target detection benchmark . journal Journal of Visual Communication and Image Representation volume 34 , pages 187--203 . https://www.sciencedirect.com/science/article...

    2016 doi

  21. [29]

    , author DeTone, D

    author Sarlin, P.E. , author DeTone, D. , author Malisiewicz, T. , author Rabinovich, A. , year 2020 . title Superglue: Learning feature matching with graph neural networks , in: booktitle Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. ...

    2020

  22. [30]

    , author Willcocks, C.G

    author Sasaki, H. , author Willcocks, C.G. , author Breckon, T. , year 2021 . title Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models . journal ArXiv volume abs/2104.05358 . https://api.semanticscholar.org/CorpusID:233210328

    2021 arXiv

  23. [31]

    , author Lew, J

    author Song, S. , author Lew, J. , author Jang, H. , author Yoon, S. , year 2024 . title Unsupervised homography estimation on multimodal image pair via alternating optimization . journal arXiv preprint arXiv:2411.13036

    2024 arXiv

  24. [32]

    , author Guo, W

    author Wei, X. , author Guo, W. , author Yu, W. , author Wei, F. , author Li, D. , year 2025 . title Osdm-mreg: Multimodal image registration based one step diffusion model . journal arXiv preprint arXiv:2504.06027

    2025

  25. [33]

    , author Tao, R

    author Xiang, Y. , author Tao, R. , author Wang, F. , author You, H. , author Han, B. , year 2020 . title Automatic registration of optical and sar images via improved phase congruency model . journal IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sen...

    2020

  26. [34]

    , author Pu, Z

    author Xiao, F. , author Pu, Z. , author Chen, J. , author Gao, X. , year 2024 a. title Dgfnet: Depth-guided cross-modality fusion network for rgb-d salient object detection . journal IEEE Transactions on Multimedia volume 26 , pages 2648--2658 . https://api.semanticscholar.or...

    2024

  27. [35]

    , author Zhang, N

    author Xiao, J. , author Zhang, N. , author Tortei, D. , author Loianno, G. , year 2024 b. title Sthn: Deep homography estimation for uav thermal geo-localization with satellite imagery . journal IEEE Robotics and Automation Letters volume 9 , pages 8754--8761 . https://api.se...

    2024

  28. [36]

    , author Ma, J

    author Xu, H. , author Ma, J. , author Yuan, J. , author Le, Z. , author Liu, W. , year 2022 . title Rfnet: Unsupervised network for mutually reinforcing multi-modal image registration and fusion . journal 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CV...

    2022

  29. [37]

    , author Tang, T

    author Ye, Y. , author Tang, T. , author Zhu, B. , author Yang, C. , author Li, B. , author Hao, S. , year 2022 . title A multiscale framework with unsupervised learning for remote sensing image registration . journal IEEE Transactions on Geoscience and Remote Sensing volume 6...

    2022

  30. [38]

    , author Cao, S.Y

    author Yu, J. , author Cao, S.Y. , author Zhang, R. , author Zhang, C. , author Hu, J. , author Yu, Z. , author Yu, B. , author Shen, H.l. , year 2024 . title Internet: Unsupervised cross-modal homography estimation based on interleaved modality transfer and self-supervised ho...

    2024 arXiv

  31. [39]

    , author Wang, C

    author Zhang, J. , author Wang, C. , author Liu, S. , author Jia, L. , author Ye, N. , author Wang, J. , author Zhou, J. , author Sun, J. , year 2020 . title Content-aware unsupervised deep homography estimation , in: booktitle European Conference on Computer Vision , organiza...

    2020

  32. [40]

    , author Ma, J

    author Zhang, K. , author Ma, J. , year 2024 . title Sparse-to-dense multimodal image registration via multi-task learning , in: editor Salakhutdinov, R. , editor Kolter, Z. , editor Heller, K. , editor Weller, A. , editor Oliver, N. , editor Scarlett, J. , editor Berkenkamp, ...

    2024

  33. [41]

    , author Ma, J

    author Zhang, R. , author Ma, J. , author Cao, S.Y. , author Luo, L. , author Yu, B. , author Chen, S.J. , author Li, J. , author Shen, H.L. , year 2025 . title Scpnet: Unsupervised cross-modal homography estimation via intra-modal self-supervised learning , in: booktitle Euro...

    2025

  34. [42]

    , author Huang, X

    author Zhao, Y. , author Huang, X. , author Zhang, Z. , year 2021 . title Deep lucas-kanade homography for multimodal image alignment , in: booktitle Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. pages 15950--15959

    2021

  35. [43]

    , author Huang, J

    author Zhou, M. , author Huang, J. , author Yan, K.R. , author Hong, D. , author Jia, X. , author Chanussot, J. , author Li, C. , year 2024 . title A general spatial-frequency learning framework for multimodal image fusion. journal IEEE transactions on pattern analysis and mac...

    2024

  36. [44]

    , author Cao, S.Y

    author Zhu, H. , author Cao, S.Y. , author Hu, J. , author Zuo, S. , author Yu, B. , author Ying, J. , author Li, J. , author Shen, H.L. , year 2024 . title Mcnet: Rethinking the core ingredients for accurate and efficient homography estimation , in: booktitle Proceedings of t...

    2024

  37. [45]

    , author Park, T

    author Zhu, J.Y. , author Park, T. , author Isola, P. , author Efros, A.A. , year 2017 . title Unpaired image-to-image translation using cycle-consistent adversarial networks , in: booktitle Computer Vision (ICCV), 2017 IEEE International Conference on

    2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.