{"id":"aee9e58c-0606-4726-aca6-967234a18fe6","arxiv_id":"2505.06920","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"B-SR is a self-supervised bi-directional registration framework for misaligned infrared-visible image fusion that uses random patch transformations and inverse consistency to train alignment, reporting top scores on RoadScene, DroneVehicle, and TNO.","lead":"This paper introduces a self-supervised method, B-SR, that aligns and fuses infrared and visible images that are spatially misaligned, without requiring manually aligned training pairs. It randomly rearranges image patches and forces the network to produce consistent alignment results, then reports the best fusion-quality scores on three benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The registration claim rests on unvalidated self-supervision and fusion-quality proxies; no direct alignment error is reported, so the PDG/IPDG signal may not be driving the reported gains.","rationale":"The reader's conditional verdict is appropriate: the experimental comparison is broad and the robustness sweeps in Section 7.3 are useful evidence, but the core mechanism is not directly validated. My reading sharpens the concern in two ways. First, Algorithm 1's notation P(V),P(T) raises a concrete implementation question: if PDG is applied independently to the infrared and visible images, the IPDG-inverted pseudo-global differences cannot be expected to equal the original global differences, so the main self-supervision loss would have an incorrect target. Second, even with a shared transform, consistency losses are vulnerable to degenerate identity solutions; the paper does not show that the learned deformation fields are nontrivial or correct. The absence of any direct registration-error metric means the fusion-quality numbers are the only evidence for the registration part of the central claim, and those numbers are known to be sensitive to sharpness and edge content rather than geometric fidelity. These issues are not accusations of error; they are testable gaps. The proposed deformation-field evaluation on synthetic data with known shifts would settle whether the registration is real. If it passes, the conditional verdict can be upgraded; if it fails, the central claim is not supported.","tokens_in":14516,"tokens_out":10529,"duration_ms":110134,"concrete_test":"On the RoadScene synthetic benchmark with a known 5-pixel horizontal shift, extract the predicted deformation field phi_p from the trained B-SR and compute the mean end-point error against the known constant shift, plus the residual displacement of the warped infrared image relative to the visible image at corresponding pixels; repeat for the dilation/center-cropping protocol of Section 7.1. If the residual registration error is not substantially below the input misalignment, or if removing the PDG/IPDG loss leaves both the fusion metrics and registration error unchanged, the self-supervision is not the source of the reported gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that B-SR improves registration and fusion under misalignment, with the PDG/IPDG self-supervision cited as the mechanism that supplies alignment constraints without ground truth. For this to be true, two conditions must hold: the registration network must be approximately equivariant to the patch flips, rotations, and stitching used by PDG, and the same patch transform must be applied jointly to both modalities so that the IPDG-inverted branch is comparable to the original branch. Neither condition is verified. Algorithm 1 writes Vnew,Tnew = P(V),P(T), which suggests independent random transforms; if the two modalities receive different transforms, the pseudo-global difference recovered after IPDG is not the original global difference and the inter-branch consistency loss optimizes an incorrect target. Even if the transforms are shared, an identity-deformation solution satisfies the inter-branch consistency loss, so the loss alone cannot certify that spatial alignment has occurred. The only quantitative evidence for alignment is the six fusion metrics Qabf, VIFF, SF, AG, MG, and EI, none of which measures geometric registration error; these metrics can improve from increased sharpness or contrast without reducing misalignment. No direct registration error, residual-displacement statistics, or significance testing is reported anywhere. The registration component of the headline claim is therefore underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes B-SR, a self-supervised bi-directional registration and fusion framework for misaligned infrared-visible image pairs. It uses a proxy data generator (PDG) and its inverse (IPDG) to construct inter-branch self-supervision, bi-directional deformation fields for intra-branch alignment, and a neighborhood dynamic alignment loss to handle the modality gap without explicit image translation. The method is evaluated on RoadScene, DroneVehicle, and TNO datasets with six fusion metrics, reporting the highest scores on all metrics in all datasets. The authors also provide ablation studies and robustness experiments under simulated dilation and center-cropping misalignment.","tokens_in":14767,"tokens_out":4195,"duration_ms":44581,"significance":"If the claims hold, B-SR addresses a practical problem: fusion of misaligned multi-modal data without ground-truth registration, and it does so without relying on image translation, which the paper argues avoids noise from translation and model dependence. The design is interesting and the experimental scope covers both simulated and real misalignment. The promise of public code is a positive factor for reproducibility. However, the registration claim is not directly verified: the paper never reports registration error, and the self-supervision target may be ill-posed as written. The fusion metrics alone are indirect evidence of alignment and can improve through contrast or sharpness effects. The central contribution needs stronger validation before it can be accepted.","major_comments":[{"comment":"The edge pixel retention loss Lepr = (Ta − V)^2 − (T − V)^2 is problematic as written. Its gradient with respect to Ta is 2(Ta − V), so minimizing it drives the aligned infrared image Ta toward the visible image V, erasing modality-specific infrared information. This contradicts the stated purpose (preserving structural differences) and would be a load-bearing flaw if the loss is implemented as in Eq. (7). Please either correct the formula, clarify the intended variable for Ta, or provide a derivation showing why this loss does not cause modality collapse.","section":"Section 3.4, Eq. (7)"},{"comment":"Algorithm 1 writes 'Vnew,Tnew = P(V),P(T)', which suggests that the stochastic PDG transform is applied independently to V and T. But the inter-branch consistency loss requires that IPDG invert the same spatial transform that was applied to both modalities; otherwise the pseudo-global differences do not correspond to the original global difference, and the self-supervision target is incorrect. The paper must state explicitly that a single shared random transform (same patch partition, flips, rotations, stitching) is used for both images, and implement it as a joint operation. If independent transforms are intended, the method needs a different justification.","section":"Algorithm 1 and Section 3.1"},{"comment":"No direct registration error is reported anywhere. For the RoadScene experiment the misalignment is a known 5-pixel horizontal shift, so the displacement residual (e.g., mean endpoint error or EPE of the deformation field) is computable and would directly validate the registration module. The six fusion metrics used (Qabf, VIFF, SF, AG, MG, EI) are quality measures that can improve from increased contrast or sharpness without geometric alignment. Without a direct alignment metric, the claim in Section 5 that B-SR 'improves the registration' is not supported by the quantitative evidence.","section":"Sections 4.1 and 4.2"},{"comment":"All quantitative results are reported as single point estimates without standard deviations, confidence intervals, or significance tests across repeated runs or test splits. Some comparisons are very close (e.g., Qabf on DroneVehicle: B-SR 0.56 vs SuperFusion 0.53; VIFF on TNO: B-SR 0.75 vs CDDFuse 0.74). Given that the paper's central claim is consistent superiority, the lack of statistical evidence is a concern that should be addressed, for instance by reporting mean and standard deviation over at least three runs or by performing a paired significance test.","section":"Tables 1-3 and Section 4.2"}],"minor_comments":[{"comment":"There are several typographical errors: 'singer or multiple alignment modules' should be 'single or multiple', and the title contains 'Self-Registrationtion' in the running text. Please proofread the manuscript.","section":"Section 1, Related Work"},{"comment":"Equation (12) appears to have a missing parenthesis: the denominator 'cc(Fl_T,(Fl_V) + epsilon' is not syntactically well-formed. Please fix the notation.","section":"Section 3.4, Eq. (12)"},{"comment":"The column headers in Table 4 ('Ls', 'Lnda', 'JO', 'Lepr', 'SS', 'SD') are not all defined in the main text. 'Ls' and 'SD' are unclear. Please define every abbreviation in the table caption or in the ablation descriptions.","section":"Table 4"},{"comment":"In the line 'Lss(TR, TR, VR'new, TR'new, ...)' the first two arguments appear to be identical; presumably one should be the visible reconstruction. Please correct the pseudocode so it matches the described loss between global and pseudo-global branches.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a promising framework, but the two central issues — the suspicious Lepr formulation and the unclear PDG transform sharing — need to be resolved before the registration claim is credible. I recommend requiring the authors to add direct registration error measurements on the synthetic RoadScene experiments and to clarify the self-supervision construction. If the Lepr loss is indeed implemented as in Eq. (7), the method may be inadvertently minimizing a modality-collapsing objective, which would undermine the fusion results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is worth a look: replace image-translation-based alignment in infrared-visible fusion with a self-supervised consistency scheme built on a proxy data generator (PDG) and its inverse (IPDG). That is a reasonable way to dodge the noise that translation modules introduce, and the ablation study (Exp. V) suggests the self-supervision does contribute. The experimental coverage is also broad: three datasets, eight competitors, six metrics, plus a robustness sweep out to 50-pixel simulated misalignment. The reported wins are consistent, and the qualitative figures show visibly cleaner fusion on the DroneVehicle set where other methods ghost. Credit is due for the range of experiments and the promise of public code.\n\nBut the central claim—that B-SR actually registers, not just sharpens or contrast-enhances—is not proven. The six metrics (Qabf, VIFF, SF, AG, MG, EI) are fusion-quality indices; none measures geometric alignment error. On the simulated RoadScene shift you know the ground-truth displacement, and the paper never reports residual displacement, success rate, or even a simple endpoint error. Without that, better fusion scores could come from increased saliency or texture rather than true spatial alignment. The stress-test note is right: the inter-branch consistency loss can be satisfied by an identity deformation, and the equivariance assumption under random patch flips/rotations is asserted, not verified. Algorithm 1 writes 'Vnew,Tnew = P(V),P(T)'—if P is drawn independently for the two modalities, the IPDG-pseudo-global difference is not the original global difference and the whole consistency target is wrong. The paper should state clearly that the same patch transform is applied jointly to both modalities and ideally visualize a few deformation fields.\n\nThe other concrete problem is Eq. 7. As written, Lepr = (Ta−V)^2 − (T−V)^2, which pushes the aligned IR image Ta toward the visible image V's pixel values, not toward preserving layout structure. The supplementary text describes a different loss (distance between pre-alignment edge offsets and post-alignment offsets), so the main-text equation is at minimum a serious typo and at worst a load-bearing mis-specification. That needs a correction, not just a clause.\n\nMinor but worth noting: no error bars or significance tests anywhere, and the 'all six metrics' claim is stated without variance even though datasets are small enough that a single image can swing a mean.\n\nVerdict: the method and framework are plausible, the writing is honest about the lack of ground truth, and the experiments are extensive enough to justify a serious referee. But currently the alignment claim is underdetermined. A revision that adds direct registration-error evaluation on simulated shifts, fixes Eq. 7, clarifies the shared PDG transform, and either verifies the equivariance or tones down the claim would make this a solid contribution. I would engage with a revised version.","headline":"Plausible self-supervised registration idea for IR-VIS fusion, but the alignment claim is under-supported because no direct registration error is reported and the main-text Lepr loss looks mis-specified.","tokens_in":15294,"tokens_out":1557,"would_cite":false,"duration_ms":18971,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A self-supervised framework aligns misaligned infrared-visible image pairs without ground-truth correspondence or image translation, and the paper reports that it beats eight prior methods on every measured metric across three datasets.","keywords":["infrared-visible image fusion","image registration","self-supervised learning","multi-modal alignment","deformation field","proxy data generator","misaligned image pairs"],"falsifier":"Create a test set with known ground-truth displacements, such as infrared images shifted by a known amount, and compute the endpoint error of the predicted deformation fields against the true displacement. If removing the inter-branch self-supervised loss does not increase this endpoint error, the PDG/IPDG branch is not the source of alignment; conversely, if the loss stays low while endpoint error remains high, the equivariance assumption alone does not guarantee correct registration.","tokens_in":14320,"feed_emoji":"🔄","tokens_out":8401,"duration_ms":71292,"temperature":0.7,"pith_summary":"The paper proposes B-SR, a fully self-supervised framework for aligning and fusing infrared-visible image pairs that are spatially misaligned. It creates its own training signal with a proxy data generator that randomly crops, flips, rotates, and re-stitches images, and an inverse generator that undoes those operations on the model's outputs. The registration network is trained so that processing the permuted image and then un-permuting the result matches processing the original image, providing global and local alignment supervision without any ground-truth displacement. The paper reports that B-SR beats eight prior methods on all six evaluation metrics for the RoadScene, DroneVehicle, and TNO datasets, covering both simulated and real misalignment. This matters because existing alignment-based fusion methods rely on image translation, which injects noise, whereas B-SR aligns without it.","feed_headline":"Self-supervised method tops eight IR-VIS fusion baselines","feed_subtitle":"B-SR aligns misaligned IR-VIS pairs without image translation, scoring highest on all six quality metrics.","key_machinery":"The load-bearing mechanism is the proxy/inverse proxy data generator pair (PDG and IPDG). PDG cuts an image into N×N patches, records random flips and rotations per patch, and re-stitches them into a new image; IPDG applies the inverse operations to the model's predicted aligned images and deformation fields. The registration module is a U-Net that predicts bi-directional deformation fields, and a spatial transformer network warps each modality accordingly. The self-supervision couples the two branches through L1 and L2 losses on aligned images, deformation fields, and reconstructions, enforcing that registration commutes with the PDG permutation. Two additional losses address the modality gap directly: the neighborhood dynamic alignment loss (Lnda) matches edge distance and angle within a local neighborhood so that the current modality aligns to valid edges of the other modality, and the edge pixel retention loss (Lepr) preserves the pre-alignment structural differences around edges. A reconstruction decoder, trained jointly with registration, provides feature-level supervision, and the fusion module later reuses the frozen encoder.","core_discovery":"The paper's central claim is that misaligned infrared-visible image pairs can be registered in a fully self-supervised way using bi-directional deformation fields, and that this registration improves downstream fusion quality. The supervisory signal is a consistency constraint between two branches: one branch processes the original images, the other processes images that have been cut into patches, randomly flipped and rotated, and re-stitched, and then applies the inverse operation to the aligned outputs and predicted deformation fields. The paper argues that these two branches should agree, so the loss enforces global-local difference consistency. It further claims that the proposed neighborhood dynamic alignment loss aligns cross-modal edges without image translation, and that joint optimization of registration with a reconstruction module sharpens both. The empirical claim is that B-SR obtains the highest scores on all six metrics across RoadScene with simulated 5-pixel shifts, DroneVehicle with large real misalignment, and TNO with small misalignment, and that it stays robust under dilation-and-crop misalignment up to 50 pixels.","pith_inferences":["A natural extension beyond the paper is to measure the registration error directly on synthetic warps with known ground truth; this would separate alignment accuracy from fusion quality and test the equivariance assumption more directly.","The same PDG/IPDG equivariance strategy could transfer to other registration tasks where paired ground truth is scarce, such as medical image alignment across modalities or multi-sensor remote sensing, since it only requires the network to be equivariant to the chosen proxy transformations.","The paper's choice of N=2 for the PDG is empirical; a principled study of how equivariance error grows with N could turn this hyperparameter into an architecture-aware calibration, possibly with curriculum scheduling of N during training.","Because the framework imposes consistency without ground truth, it will be most convincing when paired with an explicit calibration of the equivariance error of the U-Net backbone; architectures with padding or positional biases may require explicit equivariance guarantees."],"forward_implications":["If B-SR is correct, multi-modal registration for fusion no longer needs ground-truth alignment or image translation, removing a major source of noise in prior pipelines.","The reported across-the-board gains on six metrics imply that alignment quality directly translates into measurable fusion quality on both simulated and real misaligned data.","The robustness experiments suggest the method can tolerate large unknown misalignment amplitudes, such as drone-view footage, without retraining per magnitude.","The ablation results indicate that each component—neighborhood dynamic alignment, joint registration-reconstruction, bi-directional fields, and inter-branch self-supervision—contributes to the final improvement, which means removing any one of them should degrade fusion quality."],"supporting_citations":[{"why":"Grounds the self-supervision design: deep learning models have feature symmetry invariance, so registration should commute with the proxy transformations.","marker":"[35]"},{"why":"Supplies the U-Net backbone that predicts the bi-directional deformation fields.","marker":"[36]"},{"why":"Provides the spatial transformer used to warp images by the predicted deformation fields.","marker":"[37]"},{"why":"Inspires the smoothing loss on deformation field gradients.","marker":"[38]"},{"why":"Supplies the autoencoder fusion architecture and the fusion loss formulation.","marker":"[30]"},{"why":"A competing unsupervised registration-fusion baseline built on cross-modality generation.","marker":"[20]"},{"why":"A competing mutually reinforcing registration and fusion baseline that B-SR must outperform.","marker":"[21]"},{"why":"Provides the aligned RoadScene benchmark on which the paper simulates 5-pixel and dilation-crop misalignment.","marker":"[40]"},{"why":"Provides the TNO benchmark with slight real misalignment.","marker":"[41]"},{"why":"Provides the DroneVehicle benchmark with large real misalignment from drone views.","marker":"[42]"}],"fun_headline_variants":["B-SR self-registers misaligned IR-VIS pairs with no translation","Self-supervised B-SR wins on all six IR-VIS fusion metrics","Bi-directional consistency aligns IR-VIS edges without translation","B-SR: self-supervised IR-VIS alignment beats eight baselines","No ground truth needed: B-SR bi-directionally registers IR-VIS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training signal depends on the registration network being equivariant to the proxy data generator's random patch flips, rotations, and re-stitching: processing a permuted image and then undoing the permutation must equal processing the original image, and if the loss can be satisfied without true spatial alignment, the self-supervision gives no correctness signal.","fun_headline_variants_meta":{"raw":{"variants":["B-SR self-registers misaligned IR-VIS pairs with no translation","Self-supervised B-SR wins on all six IR-VIS fusion metrics","Bi-directional consistency aligns IR-VIS edges without translation","B-SR: self-supervised IR-VIS alignment beats eight baselines","No ground truth needed: B-SR bi-directionally registers IR-VIS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000343,"raw_usage":{"total_tokens":1897,"prompt_tokens":967,"completion_tokens":930,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":832}},"tokens_in":583,"tokens_out":930,"duration_ms":8577,"temperature":1.0,"reasoning_tokens":832,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:29:27.263409+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Create a test set with known ground-truth displacements, such as infrared images shifted by a known amount, and compute the endpoint error of the predicted deformation fields against the true displacement. If removing the inter-branch self-supervised loss does not increase this endpoint error, the PDG/IPDG branch is not the source of alignment; conversely, if the loss stays low while endpoint error remains high, the equivariance assumption alone does not guarantee correct registration.","supporting_citations":[{"cited_title":"Imaging with equivariant deep learning: From unrolled network design to fully unsupervised learning.IEEE Signal Processing Magazine, 40(1):134–147, 2023","cited_arxiv_id":null,"evidence_quote":"Grounds the self-supervision design: deep learning models have feature symmetry invariance, so registration should commute with the proxy transformations."},{"cited_title":"Spatial transformer networks.NIPS, 28, 2015","cited_arxiv_id":null,"evidence_quote":"Provides the spatial transformer used to warp images by the predicted deformation fields."},{"cited_title":"V oxelmorph: a learning framework for deformable medical image registration.IEEE TMI, 38(8):1788–1800, 2019","cited_arxiv_id":null,"evidence_quote":"Inspires the smoothing loss on deformation field gradients."},{"cited_title":"Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion","cited_arxiv_id":null,"evidence_quote":"Supplies the autoencoder fusion architecture and the fusion loss formulation."},{"cited_title":"Murf: Mutually reinforcing multi-modal image registration and fusion.IEEE TPAMI, 45(10):12148–12166, 2023","cited_arxiv_id":null,"evidence_quote":"A competing mutually reinforcing registration and fusion baseline that B-SR must outperform."},{"cited_title":"Fusiondn: A unified densely connected network for image fusion","cited_arxiv_id":null,"evidence_quote":"Provides the aligned RoadScene benchmark on which the paper simulates 5-pixel and dilation-crop misalignment."},{"cited_title":"Progress in color night vision.Optical Engineering, 51(1):010901– 010901, 2012","cited_arxiv_id":null,"evidence_quote":"Provides the TNO benchmark with slight real misalignment."},{"cited_title":"Drone-based rgb-infrared cross-modality vehicle detection via uncertainty-aware learning.IEEE TCSVT, 32(10):6700–6713, 2022","cited_arxiv_id":null,"evidence_quote":"Provides the DroneVehicle benchmark with large real misalignment from drone views."}],"review_version":1}