Pith. sign in

REVIEW 4 major objections 5 minor 69 references

MINIMA: Modality Invariant Image Matching

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A data engine turns cheap RGB pairs into a large synthetic multimodal dataset with valid labels, and fine-tuning existing matchers on it yields one model that generalizes to real cross-modal matching, including unseen modality pairs.

desk verdict Worth a serious referee: the data engine and MD-syn are a real contribution, but the universal-superiority claims need trimming and the event label issue needs a fix. read the letter →

arxiv 2412.19412 v2 pith:VMNUYNYN submitted 2024-12-27 cs.CV

Jiangwei Ren , Xingyu Jiang , Zizhuo Li , Dingkang Liang , Xin Zhou , Xiang Bai This is my paper · ORCID
classification cs.CV
keywords cross-modalimagematchingsyntheticdatagenerationmodalityinvarianceengineMD-syndatasetzero-shotgeneralizationfine-tuningmodelsmultimodalperception
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MINIMA's thesis is that cross-modal image matching fails for lack of data, not for lack of model design, and that the missing data can be synthesized. The paper builds a data engine that takes RGB image pairs from MegaDepth, which already have dense depth and pose labels, and generates six pseudo-modality versions of each image (infrared, depth, event, surface normal, and two artistic styles) without changing the known geometry. The resulting MD-syn dataset is used to fine-tune standard matchers that were pre-trained on RGB, yielding one model per backbone that handles many modality pairs. The paper reports large gains on real RGB-infrared, RGB-depth, and RGB-event benchmarks and on zero-shot remote sensing and medical cases, often beating methods trained specifically for one modality pair. A sympathetic reading: if the transfer holds, real multimodal capture and manual labeling can be replaced by generation from abundant RGB data.

What carries the argument

MD-syn, a synthetic multimodal dataset produced by a data engine of six generative models applied per image to MegaDepth RGB pairs. Because each generator transforms a single view in place, the inherited depth and pose labels remain geometrically consistent for all generated pairs. The training recipe pairs this dataset with a two-stage procedure: pre-train on RGB MegaDepth, then fine-tune on randomly sampled cross-modal pairs, so the final model is not specialized to any one modality pair.

What would settle it

Measure geometric consistency of generated pairs directly: warp the generated modality image of the second view into the first view using the inherited MegaDepth depth and pose; if the reprojection error is large for event or depth modalities, the labels are corrupted. Alternatively, train on MD-syn with the event generator's random motion disabled and on a depth generator with known metric-scale error; if zero-shot AUC on real DSEC and DIODE does not fall, the reported gains do not depend on the assumed geometry preservation.

Watch

Extended reading notes

Core claim

The paper's central claim is that modality invariance can be acquired from synthetic scale: fine-tuning an off-the-shelf matcher on randomly selected pairs of generated pseudo-modalities, all inheriting MegaDepth geometry, produces a single model that matches real images across different sensors. The data engine applies per-image generators, so the relative pose and depth computed once for each RGB pair remain valid labels for every generated cross-modal pair, which is what makes label inheritance sound. Using only RGB-IR, RGB-depth, and RGB-normal synthetic pairs for fine-tuning, the resulting MINIMA models transfer to real RGB-IR, RGB-depth, RGB-event, remote sensing, and medical matching across 19 cross-modal cases, with the dense RoMa variant showing the largest improvements. The paper attributes the gains to the data engine rather than to any new matching architecture.

Load-bearing premise

The generated pseudo-modalities preserve the geometry of the source RGB pairs and reproduce the appearance gap of real cross-modal sensors closely enough that the inherited matching labels stay valid and training transfers to real cameras.

Editorial extensions

If this is right

  • One matcher fine-tuned on MD-syn can be deployed on many cross-modal tasks (RGB-IR, RGB-depth, RGB-event, optical-SAR, medical pairs) without per-modality retraining.
  • Synthetic-only training transfers to real sensors, with the reported pose-estimation AUC on METU-VisTIR increasing by over 400% for the sparse variant and similar gains on DIODE and DSEC.
  • Adding a new modality reduces to adding a generator to the engine, so the dataset grows without any new real multimodal capture.
  • Fine-tuning on a mix of synthetic modalities outperforms fine-tuning on a single one, and the models mostly retain their RGB-only matching accuracy on MegaDepth-1500, with LoFTR being the exception.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method implies that the dominant cost of building cross-modal matchers shifts from sensor collection to generator fidelity, so improving style transfer and depth estimation should translate directly into matching gains.
  • A natural stress test is to vary generator realism while holding the training pipeline fixed; if real-world AUC tracks generator fidelity, the synthetic-to-real transfer is driven by the modality gap statistics rather than by unrelated priors.
  • The same label-inheritance recipe could generate pseudo-modal pairs for other geometric tasks such as optical flow or stereo, where real cross-modal ground truth is even scarcer.
  • The retained RGB-only performance (except LoFTR) suggests the fine-tuning acts partly as a regularizer, hinting that new modalities could be added incrementally without catastrophic forgetting.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MINIMA, a unified cross-modal image matching framework built on a synthetic data engine. Starting from MegaDepth RGB pairs with known depth/pose, the authors generate pseudo-modalities (Infrared, Depth, Event, Normal, and two artistic styles) using off-the-shelf or fine-tuned generative models, producing MD-syn with over 480M cross-modal pairs. They then fine-tune existing matchers (LightGlue, LoFTR, RoMa) on randomly selected modality pairs and evaluate on synthetic and real datasets, including zero-shot medical, remote sensing, and RGB-Event cases. The central claim is that a single matcher fine-tuned only on synthetic multimodal data can generalize across many real cross-modal matching tasks and can outperform modality-specific methods.

Significance. If the central claim holds, the paper makes a valuable empirical contribution: it offers a scalable recipe for cross-modal matching data without expensive multi-sensor capture, and it demonstrates gains on several real benchmarks across three matcher families. The paper is also commendable for the scale of its evaluation, its ablation studies, and the stated intent to release dataset and code. However, the validity of the claimed universal superiority depends on two load-bearing assumptions: that all generated pseudo-modalities inherit MegaDepth geometry labels exactly, and that the data-cleaning step does not circularly bias the training/evaluation. The event-modality generation appears to violate the first assumption, and the cleaning step in Appendix A.2 is described in a way that invites circularity concerns. With those points addressed, the empirical evidence for the IR/Depth/Normal cases would support publication.

major comments (4)
  1. [Sec. 3.3, Event paragraph, and Tab. 2] The event simulation is internally inconsistent with the claimed label inheritance. After Eq. (1), the authors state that they "give a random slight motion to compute the event responses." An event image generated this way records brightness changes along the trajectory of the injected random motion, not the projection of the scene point corresponding to the RGB pixel at the same location. Therefore the MegaDepth pose/depth labels for RGB-Event pairs in MD-syn are not geometrically valid, and the synthetic RGB-Event AUC values in Tab. 2 are not a valid accuracy measure. This directly affects the Sec. 3.1 claim that labels are "directly inherited" and the "19 cross-modal cases" count in the abstract. Please either generate events using the known relative motion between the two MegaDepth views without an extra random motion field, or remove event from the label-validity claims and re-evaluate the affected tables.
  2. [Appendix A.2, Data Cleaning] The data-cleaning step uses "our matching model (fine-tuned on the target modality)" to filter training pairs by recovery of the identity homography. This is circular if the model used for cleaning was trained on the same pairs, or if the final models are evaluated on the cleaned set without reporting results on the uncleaned set. Even though only 0.91% of pairs are dropped, the procedure should specify exactly which model is used, when it was trained relative to the cleaning step, and whether the same filter is applied to test pairs. Without this, the cleaned MD-syn numbers may reflect selection toward easy examples for the chosen matcher rather than intrinsic data quality.
  3. [Abstract and Sec. 5.3, Tabs. 3 and 4] The claim that MINIMA "can significantly outperform the baselines and even surpass modality-specific methods" is too broad as stated. In Tab. 3, MINIMALoFTR achieves AUC@10 of 30.84 on real RGB-IR, below XoFTR's 34.64. In Tab. 4, MINIMALoFTR achieves 18.65 AUC@5px on real RGB-Depth, below XoFTR's 27.24. The superiority claim is defensible for the LightGlue and RoMa pipelines on several datasets, but not for all pipelines and datasets. Please qualify the claim by pipeline category and dataset, or restrict the abstract and conclusion to the supported cases.
  4. [Sec. 3.3, Depth and Normal generation] For Depth and Normal, the paper uses monocular estimators (DepthAnything V2 and DSINE) whose outputs are not guaranteed to be geometrically consistent with the MegaDepth camera poses or metric scale. Since RGB-Depth and RGB-Normal are two of the three modality pairs used in the final training (Sec. 5.1), the inherited-label assumption is load-bearing for these modalities as well. Unlike the infrared generator, which is validated in Appendix A.1, there is no quantitative validation of geometric consistency for depth/normal outputs. Please provide a direct comparison of generated depth/normal against MegaDepth ground-truth geometry on a held-out split, or otherwise demonstrate that label errors are small enough to be negligible.
minor comments (5)
  1. [Sec. 5.1 and Tab. 5] The RGB-Event zero-shot evaluation uses only 100 pairs from DSEC; please state whether the reported AUCs have confidence intervals, since several method differences in Tab. 5 are only a few percentage points.
  2. [Fig. 1 caption] The caption says "Six Datasets of Real Cross-modal Image Pairs," but the left panel includes results from the synthetic MD-syn test split; please clarify which panels are synthetic and which are real.
  3. [Sec. 5.1 and Tab. A2] The text says the final model uses only RGB-IR, RGB-Depth, and RGB-Normal pairs, but Tab. A2 also reports models trained with Event, Paint, and Sketch data; please clarify which rows of Tab. A2 correspond to the models used in the main tables.
  4. [Appendix C, Figures A5-A8] Several appendix figure captions are duplicated with different dataset citations (e.g., Figure A5 appears twice, once citing METU-VisTIR [45] and once [43]); please deduplicate and correct the citations.
  5. [General] There are minor capitalization and reference inconsistencies, such as "Megadepth-1500" in Tab. A5 and inconsistent citation numbers for M3FD/PIAFusion; please proofread the final version.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor self-filtering in MD-syn data cleaning; central zero-shot claims remain independently grounded.

  1. self definitional [Appendix A.2 (Data Cleaning)]
    "To this end, and for each RGB image and its corresponding pseudo modalities, we use our matching model (fine-tuned on the target modality) to recover the homographies (the ground truths are the identity matrix) for them. Any image pair with the mean projection error of corner points larger than 10 pixels is regarded as dirty data and dropped."

    The quality criterion for a training pair is whether the matching model (already fine-tuned on the same target modality) can recover the identity homography. Pairs the model fails on are discarded, so the surviving MD-syn training set is, by construction, the subset on which the model already performs well. Reporting synthetic MD-syn results (Tab. 2) after this filter partly measures the model's own selection bias rather than an independent property of the data engine. The effect is small (0.91% of pairs dropped) and the real-data zero-shot evaluations are not filtered this way, so this step is not load-bearing for the central claim.

full rationale

The central derivation is self-contained: MD-syn is constructed by applying fixed generative models (DepthAnything V2, DSINE, StyleBooth, etc.) to MegaDepth RGB pairs, with geometric labels inherited from the RGB source. Training is standard fine-tuning of LightGlue, LoFTR, and RoMa on the resulting multimodal pairs, and the headline claims are tested on held-out real datasets (METU-VisTIR, DIODE, DSEC, MMIM) that are not used as training data for the matching models. These zero-shot evaluations provide independent support and do not reduce to the paper's own inputs. The only genuinely circular step is the Appendix A.2 data-cleaning filter, which uses a model fine-tuned on the target modality to drop its own training pairs; because only 0.91% of pairs are dropped and the main conclusions are corroborated on real sensor data, this is a minor methodological blemish rather than a collapse of the derivation. The event-modality random-motion procedure raises a label-validity concern (a randomly moved source image may not share MegaDepth geometry), but that is a correctness issue, not a circularity, so it does not increase the circularity score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the fidelity of synthetic modality generation and on the transferability of synthetic training data to real sensors. These are domain assumptions about distribution shift and label validity, not fitted constants.

free parameters (3)
  • Event simulation contrast threshold C = random in [0.05, 0.5]
    Chosen by hand to simulate varied event sensors; affects density and difficulty of generated RGB-Event pairs, a modality used only in zero-shot evaluation.
  • Projection error threshold for data cleaning = 10 px
    Pairs with mean corner projection error above 10 px are dropped; this arbitrary threshold controls training data quality and is applied using a model trained on the same data.
  • Number of generated cross-modal pairs = 480M (K=6 modalities x 40M RGB pairs x 2 directions)
    Scale of MD-syn is a design choice; the paper argues more data helps, but the exact scale is not ablated.
assumptions (4)
  • domain assumption Generated pseudo-modalities inherit the accurate geometry and matching labels of source RGB pairs.
    Invoked in Sec. 3.1 when claiming labels are directly inherited; if style transfer or modality synthesis distorts geometry, the dense training labels are invalid.
  • domain assumption The distribution shift between RGB and each generated pseudo-modality is representative of real RGB-to-sensor shifts.
    Sec. 5.6 discusses the real-pseudo gap but asserts it has little impact; the entire zero-shot transfer to METU-VisTIR, DIODE, DSEC, and MMIM depends on this.
  • domain assumption MegaDepth's COLMAP-derived depth and pose labels are sufficiently accurate for training matching models.
    Used as the source of all geometric labels in MD-syn; inherited from MegaDepth without re-verification.
  • domain assumption The generative models (StyleBooth, DepthAnything V2, DSINE, Paint Transformer, Anime2Sketch) produce outputs whose appearance matches the target real modality.
    The data engine relies on these off-the-shelf models; their fidelity is only spot-checked for infrared (Appendix A.1), not for depth, normal, or artistic modalities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MINIMA: Modality Invariant Image Matching." pith.science (2026). https://pith.science/paper/VMNUYNYN

@misc{pith2026241219412,
  author       = {Pith},
  title        = {Pith review of: MINIMA: Modality Invariant Image Matching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VMNUYNYN}},
  note         = {Machine review of arXiv:2412.19412}
}
abstract

Image matching for both cross-view and cross-modality plays a critical role in multimodal perception. In practice, the modality gap caused by different imaging systems/styles poses great challenges to the matching task. Existing works try to extract invariant features for specific modalities and train on limited datasets, showing poor generalization. In this paper, we present MINIMA, a unified image matching framework for multiple cross-modal cases. Without pursuing fancy modules, our MINIMA aims to enhance universal performance from the perspective of data scaling up. For such purpose, we propose a simple yet effective data engine that can freely produce a large dataset containing multiple modalities, rich scenarios, and accurate matching labels. Specifically, we scale up the modalities from cheap but rich RGB-only matching data, by means of generative models. Under this setting, the matching labels and rich diversity of the RGB dataset are well inherited by the generated multimodal data. Benefiting from this, we construct MD-syn, a new comprehensive dataset that fills the data gap for general multimodal image matching. With MD-syn, we can directly train any advanced matching pipeline on randomly selected modality pairs to obtain cross-modal ability. Extensive experiments on in-domain and zero-shot matching tasks, including $19$ cross-modal cases, demonstrate that our MINIMA can significantly outperform the baselines and even surpass modality-specific methods. The dataset and code are available at https://github.com/LSXI7/MINIMA.

Figures

Figures reproduced from arXiv: 2412.19412 by the authors.

Figure 1
Figure 1. Overall Image Matching Accuracy and Efficiency on Six Datasets of Real Cross-modal Image Pairs. AUC of the pose error (@10◦ ) or reprojection error (@10px) is used for accu￾racy evaluation, while Pairs Per Second is used for efficiency test. Left: AUCs on each dataset of representative methods are reported. Right: average performance is summarized, wherein different col￾ors indicate matching pipelines of sparse, sem… view at source ↗
Figure 2
Figure 2. Qualitative Results on Real Cross-modal Image Pairs. Our methods MINIMALG (sparse) and MINIMARoMa (dense) are compared with the sparse matching pipeline ReDFeat [7] and OmniGlue [20], and semi-dense matcher XoFTR [43]. ReDFeat and XoFTR are cross-modal methods, and OmniGlue is known for its generalization ability. Matches generated by each method are drawn, where the red lines indicate epipolar error (pose) or proje… view at source ↗
Figure 3
Figure 3. Overview of the Proposed MINIMA Pipeline: Trained Once to Achieve Any Cross-modal Matching Tasks. Wherein the Data Engine is to generate a large multimodal matching dataset, supporting the training of matching models to obtain cross-modal ability. 3.1. Advantages of Cross-Modal Generation The ideal strategy is to capture real images of multiple modalities in the wild. But obviously, it is impractical to arrange mult… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 58 canonical work pages

  1. [1]

    Thermal voyager: A com- parative study of rgb and thermal cameras for night-time au- tonomous navigation

    NG Aditya, PB Dhruval, Jehan Shalabi, Shubhankar Jape, Xueji Wang, and Zubin Jacob. Thermal voyager: A com- parative study of rgb and thermal cameras for night-time au- tonomous navigation. In ICRA, pages 14116–14122, 2024. 1

    work page 2024

  2. [2]

    A low power, fully event-based gesture recognition system

    Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jef- frey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, et al. A low power, fully event-based gesture recognition system. In CVPR, pages 7243–7252, 2017. 3

    work page 2017

  3. [3]

    Netvlad: Cnn architecture for weakly supervised place recognition

    Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pa- jdla, and Josef Sivic. Netvlad: Cnn architecture for weakly supervised place recognition. In CVPR, pages 5297–5307,

  4. [4]

    Gwangbin Bae and Andrew J. Davison. Rethinking inductive biases for surface normal estimation. In CVPR, 2024. 5

    work page 2024

  5. [5]

    Learning to match features with seeded graph matching network

    Hongkai Chen, Zixin Luo, Jiahui Zhang, Lei Zhou, Xuyang Bai, Zeyu Hu, Chiew-Lan Tai, and Long Quan. Learning to match features with seeded graph matching network. In ICCV, pages 6301–6310, 2021. 15

    work page 2021

  6. [6]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nie10ner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, pages 5828–5839, 2017. 1, 3, 4

    work page 2017

  7. [7]

    Crosshomo: Cross-modality and cross- resolution homography estimation

    Xin Deng, Enpeng Liu, Chao Gao, Shengxi Li, Shuhang Gu, and Mai Xu. Crosshomo: Cross-modality and cross- resolution homography estimation. IEEE Trans. Pattern Anal. Mach. Intell., 2024. 2, 4

    work page 2024

  8. [8]

    Redfeat: Recoupling detection and description for multimodal feature learning.IEEE Trans

    Yuxin Deng and Jiayi Ma. Redfeat: Recoupling detection and description for multimodal feature learning.IEEE Trans. Image Process., 32:591–602, 2022. 2, 3, 4, 6, 7, 8, 15

    work page 2022

Show all 69 references
  1. [9]

    Superpoint: Self-supervised interest point detection and description

    Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabi- novich. Superpoint: Self-supervised interest point detection and description. In CVPRW, pages 224–236, 2018. 2, 13, 15

    2018

  2. [10]

    Dkm: Dense kernelized feature matching for geometry estimation

    Johan Edstedt, Ioannis Athanasiadis, M ˚arten Wadenb ¨ack, and Michael Felsberg. Dkm: Dense kernelized feature matching for geometry estimation. In CVPR, pages 17765– 17775, 2023. 1, 2, 4, 6, 7, 8, 15

    2023

  3. [11]

    Roma: Robust dense feature matching

    Johan Edstedt, Qiyu Sun, Georg B ¨okman, M ˚arten Wadenb¨ack, and Michael Felsberg. Roma: Robust dense feature matching. In CVPR, pages 19790–19800, 2024. 1, 2, 4, 5, 6, 7, 8, 13, 15

    2024

  4. [12]

    Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.Communications of the ACM, 24(6):381–395, 1981

    Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.Communications of the ACM, 24(6):381–395, 1981. 6

    1981

  5. [13]

    Event-based vision: A survey

    Guillermo Gallego, Tobi Delbr ¨uck, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, J ¨org Conradt, Kostas Daniilidis, et al. Event-based vision: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 44(1):154–180, 2020. 5

    2020

  6. [14]

    Low-latency auto- motive vision with event cameras

    Daniel Gehrig and Davide Scaramuzza. Low-latency auto- motive vision with event cameras. Nature, 629(8014):1034– 1040, 2024. 1

    2024

  7. [15]

    Video to events: Recycling video datasets for event cameras

    Daniel Gehrig, Mathias Gehrig, Javier Hidalgo-Carri ´o, and Davide Scaramuzza. Video to events: Recycling video datasets for event cameras. In CVPR, pages 3586–3595,

  8. [16]

    Stylebooth: Image style editing with mul- timodal instruction

    Zhen Han, Chaojie Mao, Zeyinzi Jiang, Yulin Pan, and Jingfeng Zhang. Stylebooth: Image style editing with mul- timodal instruction. arXiv preprint arXiv:2404.12154, 2024. 3, 5, 8, 12

    2024 arXiv

  9. [17]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. In NeurIPS, pages 6840–6851,

  10. [18]

    Pos-gift: A geomet- ric and intensity-invariant feature transformation for multi- modal images

    Zhuolu Hou, Yuxuan Liu, and Li Zhang. Pos-gift: A geomet- ric and intensity-invariant feature transformation for multi- modal images. Information Fusion, 102:102027, 2024. 3

    2024

  11. [19]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022. 5

    2022

  12. [20]

    Llvip: A visible-infrared paired dataset for low-light vision

    Xinyu Jia, Chuang Zhu, Minzhen Li, Wenqi Tang, and Wenli Zhou. Llvip: A visible-infrared paired dataset for low-light vision. In ICCV, pages 3496–3504, 2021. 1, 3, 5, 12

    2021

  13. [21]

    Omniglue: Generalizable feature match- ing with foundation model guidance

    Hanwen Jiang, Arjun Karpur, Bingyi Cao, Qixing Huang, and Andr´e Araujo. Omniglue: Generalizable feature match- ing with foundation model guidance. In CVPR, pages 19865–19875, 2024. 2, 6, 15

    2024

  14. [22]

    A review of multimodal image matching: Methods and applications

    Xingyu Jiang, Jiayi Ma, Guobao Xiao, Zhenfeng Shao, and Xiaojie Guo. A review of multimodal image matching: Methods and applications. Information Fusion, 73:22–71,

  15. [23]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. 2012. 12

    2012

  16. [24]

    Rift: Multi-modal image matching based on radiation-variation insensitive fea- ture transform

    Jiayuan Li, Qingwu Hu, and Mingyao Ai. Rift: Multi-modal image matching based on radiation-variation insensitive fea- ture transform. IEEE Trans. Image Process., 29:3296–3310,

  17. [25]

    Lnift: Locally normalized image for rotation invariant multimodal feature matching

    Jiayuan Li, Wangyi Xu, Pengcheng Shi, Yongjun Zhang, and Qingwu Hu. Lnift: Locally normalized image for rotation invariant multimodal feature matching. IEEE Trans. Geosci. Remote Sens., 60:1–14, 2022. 6, 7

    2022

  18. [26]

    Multimodal image matching: A scale-invariant algorithm and an open dataset

    Jiayuan Li, Qingwu Hu, and Yongjun Zhang. Multimodal image matching: A scale-invariant algorithm and an open dataset. ISPRS J Photogramm, 204:77–88, 2023. 6, 7

    2023

  19. [27]

    Megadepth: Learning single- view depth prediction from internet photos

    Zhengqi Li and Noah Snavely. Megadepth: Learning single- view depth prediction from internet photos. In CVPR, pages 2041–2050, 2018. 1, 3, 4, 5, 13, 14

    2018

  20. [28]

    A 128 × 128 120db 15µs latency asynchronous temporal con- trast vision sensor

    Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck. A 128 × 128 120db 15µs latency asynchronous temporal con- trast vision sensor. IEEE Journal of Solid-State Circuits, 43 (2):566–576, 2008. 5

    2008

  21. [29]

    Lightglue: Local feature matching at light speed

    Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Polle- feys. Lightglue: Local feature matching at light speed. In ICCV, pages 17627–17638, 2023. 2, 3, 4, 5, 6, 7, 8, 13, 15

    2023

  22. [30]

    Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection

    Jinyuan Liu, Xin Fan, Zhanbo Huang, Guanyao Wu, Risheng Liu, Wei Zhong, and Zhongxuan Luo. Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection. In CVPR, pages 5802–5811, 2022. 1, 3, 5, 12

    2022

  23. [31]

    Paint trans- former: Feed forward neural painting with stroke prediction

    Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Ruifeng Deng, Xin Li, Errui Ding, and Hao Wang. Paint trans- former: Feed forward neural painting with stroke prediction. In ICCV, pages 6598–6607, 2021. 5

    2021

  24. [32]

    Grid: Guided refinement for detector-free multimodal image matching

    Yuyan Liu, Wei He, and Hongyan Zhang. Grid: Guided refinement for detector-free multimodal image matching. IEEE Trans. Image Process., 2024. 3

    2024

  25. [33]

    Image matching from handcrafted to deep fea- tures: A survey

    Jiayi Ma, Xingyu Jiang, Aoxiang Fan, Junjun Jiang, and Junchi Yan. Image matching from handcrafted to deep fea- tures: A survey. Int. J. Comput. Vis., 129(1):23–79, 2021. 1, 2, 3

    2021

  26. [34]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transac. Machine Learning Research, 2023. 7

    2023

  27. [35]

    From coarse to fine: Robust hierarchical localization at large scale

    Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk. From coarse to fine: Robust hierarchical localization at large scale. In CVPR, pages 12716–12725,

  28. [36]

    Superglue: Learning feature matching with graph neural networks

    Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. InCVPR, pages 4938– 4947, 2020. 1, 2, 3, 6, 7, 8, 15

    2020

  29. [37]

    Benchmarking 6dof outdoor visual localization in changing conditions

    Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, et al. Benchmarking 6dof outdoor visual localization in changing conditions. In CVPR, pages 8601–8610, 2018. 15

    2018

  30. [38]

    Structure-from-motion revisited

    Johannes Lutz Sch ¨onberger and Jan-Michael Frahm. Structure-from-motion revisited. In CVPR, 2016. 3, 4, 15

    2016

  31. [39]

    Pixelwise view selection for un- structured multi-view stereo

    Johannes Lutz Sch ¨onberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for un- structured multi-view stereo. In ECCV, 2016. 3, 4, 15

    2016

  32. [40]

    pytorch-fid: FID Score for PyTorch

    Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. https://github.com/mseitzer/pytorch-fid ,

  33. [41]

    Gim: Learning generalizable image matcher from internet videos

    Xuelun Shen, Zhipeng Cai, Wei Yin, Matthias M ¨uller, Zijun Li, Kaixuan Wang, Xiaozhi Chen, and Cheng Wang. Gim: Learning generalizable image matcher from internet videos. In ICLR, 2024. 3, 4, 6, 7, 8, 14, 15

    2024

  34. [42]

    Indoor segmentation and support inference from rgbd images

    Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, pages 746–760. Springer, 2012. 3

    2012

  35. [43]

    Loftr: Detector-free local feature matching with transformers

    Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In CVPR, pages 8922–8931, 2021. 1, 2, 5, 6, 7, 8, 13, 14, 15

    2021

  36. [44]

    Piafusion: A progressive infrared and visible im- age fusion network based on illumination aware.Information Fusion, 2022

    Linfeng Tang, Jiteng Yuan, Hao Zhang, Xingyu Jiang, and Jiayi Ma. Piafusion: A progressive infrared and visible im- age fusion network based on illumination aware.Information Fusion, 2022. 12

    2022

  37. [45]

    Xoftr: Cross-modal feature matching transformer

    ¨Onder Tuzcuo ˘glu, Aybora K ¨oksal, Bu ˘gra Sofu, Sinan Kalkan, and A Aydin Alatan. Xoftr: Cross-modal feature matching transformer. In CVPR, pages 4275–4286, 2024. 1, 2, 3, 6, 7, 8, 12, 13, 14, 15, 16

    2024

  38. [46]

    Diode: A dense indoor and outdoor depth dataset

    Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang, Falcon Z Dai, Andrea F Daniele, Moham- madreza Mostajabi, Steven Basart, Matthew R Walter, et al. Diode: A dense indoor and outdoor depth dataset. arXiv preprint arXiv:1908.00463, 2019. 6, 7, 15, 16

    1908 arXiv

  39. [47]

    Unsuper- vised misaligned infrared and visible image fusion via cross- modality image generation and registration

    Di Wang, Jinyuan Liu, Xin Fan, and Risheng Liu. Unsuper- vised misaligned infrared and visible image fusion via cross- modality image generation and registration. In IJCAI, pages 3508–3515, 2022. 12

    2022

  40. [48]

    Visev- ent: Reliable object tracking via collaboration of frame and event flows

    Xiao Wang, Jianing Li, Lin Zhu, Zhipeng Zhang, Zhe Chen, Xin Li, Yaowei Wang, Yonghong Tian, and Feng Wu. Visev- ent: Reliable object tracking via collaboration of frame and event flows. IEEE Trans. Cybern., 2023. 6, 7, 15, 16

    2023

  41. [49]

    Efficient loftr: Semi-dense local feature matching with sparse-like speed

    Yifan Wang, Xingyi He, Sida Peng, Dongli Tan, and Xiaowei Zhou. Efficient loftr: Semi-dense local feature matching with sparse-like speed. In CVPR, pages 21666–21675, 2024. 1, 2, 4, 5, 6, 7, 8, 13, 14, 15

    2024

  42. [50]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 13(4): 600–612, 2004. 12

    2004

  43. [51]

    Single-model and any-modality for video ob- ject tracking

    Zongwei Wu, Jilai Zheng, Xiangxuan Ren, Florin-Alexandru Vasluianu, Chao Ma, Danda Pani Paudel, Luc Van Gool, and Radu Timofte. Single-model and any-modality for video ob- ject tracking. In CVPR, pages 19156–19166, 2024. 1

    2024

  44. [52]

    Adversarial open domain adap- tation for sketch-to-photo synthesis

    Xiaoyu Xiang, Ding Liu, Xiao Yang, Yiheng Zhu, Xiaohui Shen, and Jan P Allebach. Adversarial open domain adap- tation for sketch-to-photo synthesis. In WACV, pages 1434– 1444, 2022. 5

    2022

  45. [53]

    Murf: Mutually reinforc- ing multi-modal image registration and fusion

    Han Xu, Jiteng Yuan, and Jiayi Ma. Murf: Mutually reinforc- ing multi-modal image registration and fusion. IEEE Trans. Pattern Anal. Mach. Intell., 45(10):12148–12166, 2023. 1, 8

    2023

  46. [54]

    Towards grand unified representation learning for unsupervised visible-infrared per- son re-identification

    Bin Yang, Jun Chen, and Mang Ye. Towards grand unified representation learning for unsupervised visible-infrared per- son re-identification. In ICCV, pages 11069–11079, 2023. 1

    2023

  47. [55]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, pages 10371–10381, 2024. 2, 3

    2024

  48. [56]

    Depth any- thing v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2. In NeurIPS, 2024. 3, 5

    2024

  49. [57]

    Multi-modal remote sensing image matching considering co-occurrence filter

    Yongxiang Yao, Yongjun Zhang, Yi Wan, Xinyi Liu, Xiaohu Yan, and Jiayuan Li. Multi-modal remote sensing image matching considering co-occurrence filter. IEEE Trans. Im- age Process., 31:2584–2597, 2022. 3

    2022

  50. [58]

    Fast and robust matching for multimodal re- mote sensing image registration

    Yuanxin Ye, Lorenzo Bruzzone, Jie Shan, Francesca Bovolo, and Qing Zhu. Fast and robust matching for multimodal re- mote sensing image registration. IEEE Trans. Geosci. Re- mote Sens., 57(11):9059–9070, 2019. 3

    2019

  51. [59]

    Image fusion meets deep learning: A survey and perspective

    Hao Zhang, Han Xu, Xin Tian, Junjun Jiang, and Jiayi Ma. Image fusion meets deep learning: A survey and perspective. Information Fusion, 76:323–336, 2021. 1

    2021

  52. [60]

    Sparse-to-dense multimodal image registration via multi-task learning

    Kaining Zhang and Jiayi Ma. Sparse-to-dense multimodal image registration via multi-task learning. In ICML, 2024. 2, 4

    2024

  53. [61]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586–595,

  54. [62]

    Convmatch: Rethinking net- work design for two-view correspondence learning

    Shihua Zhang and Jiayi Ma. Convmatch: Rethinking net- work design for two-view correspondence learning. IEEE Trans. Pattern Anal. Mach. Intell., 2023. 15

    2023

  55. [63]

    Diverse embedding ex- pansion network and low-light cross-modality benchmark for visible-infrared person re-identification

    Yukang Zhang and Hanzi Wang. Diverse embedding ex- pansion network and low-light cross-modality benchmark for visible-infrared person re-identification. In CVPR, pages 2153–2162, 2023. 1

    2023

  56. [64]

    Self- supervised pretraining via multimodality images with trans- former for change detection

    Yuxiang Zhang, Yang Zhao, Yanni Dong, and Bo Du. Self- supervised pretraining via multimodality images with trans- former for change detection. IEEE Trans. Geosci. Remote Sens., 61:1–11, 2023. 1

    2023

  57. [65]

    Vm- loc: Variational fusion for learning-based multimodal cam- era localization

    Kaichen Zhou, Changhao Chen, Bing Wang, Muhamad Risqi U Saputra, Niki Trigoni, and Andrew Markham. Vm- loc: Variational fusion for learning-based multimodal cam- era localization. In AAAI, pages 6165–6173, 2021. 1

    2021

  58. [66]

    Visual prompt multi-modal tracking

    Jiawen Zhu, Simiao Lai, Xin Chen, Dong Wang, and Huchuan Lu. Visual prompt multi-modal tracking. InCVPR, pages 9516–9526, 2023. 1 Appendices We first provide more details of our data engine and the pro- posed MINIMA. Then we conduct additional experiments, including more ablat...

    2023

  59. [68]

    XoFTR used a handcrafted method to transfer RGB to IR, while CPSTN is a cycle-consistent perceptual network

    [45] and CPSTN (IJCAI 22) [47] as baseline methods. XoFTR used a handcrafted method to transfer RGB to IR, while CPSTN is a cycle-consistent perceptual network. We employ quantitative metrics including PSNR (Peak Signal- to-Noise Ratio), SSIM (Structural Similarity Index Mea- ...

  60. [69]

    Bold indicates the best

    [45] are used for comparison. Bold indicates the best. Data Method PSNR ↑ SSIM ↑ LPIPS ↓ FID-2048 ↓ LLVIP CPSTN 27.91 0.32 0.66 303.55 XoFTR 27.90 0.29 0.71 204.44 Ours 28.28 0.55 0.42 145.93 M3FD CPSTN 27.82 0.37 0.56 161.71 XoFTR 27.86 0.33 0.59 125.07 Ours 28.14 0.53 0.46 1...

  61. [2021]

    1, 3, 4, 6, 7, 13, 15, 17

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.