Pith. sign in

REVIEW 4 major objections 5 minor 23 references

Multispectral Detection Transformer with Infrared-Centric Feature Fusion

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that infrared should be the primary modality in multispectral object detection, with a lightweight RGB branch providing supporting semantic context, and that this asymmetric design reaches state-of-the-art accuracy at…

desk verdict Real architecture and public code, but the paper's central 'IR-centric beats symmetric' claim is contradicted by its own Table III, so it needs major revision before the empirical story is credible. read the letter →

arxiv 2505.15137 v2 pith:WEUEWHMO submitted 2025-05-21 cs.CV

Seongmin Hwang , Daeyoung Han , Moongu Jeon This is my paper · ORCID
classification cs.CV
keywords multispectralobjectdetectioninfrared-centricfusiontransformercross-modalgatingfeaturedistillationRT-DETRLLVIPFLIR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that infrared is the modality that should carry most of the detection load in multispectral object detection, with RGB playing a supporting role. The authors support this with wavelet decompositions showing that IR sub-bands carry sharp object boundaries, and with unimodal FLIR results where IR-only detectors beat RGB-only detectors. On that basis they build IC-Fusion, a detection transformer pairing a lightweight ResNet-18 RGB stream with a deeper ResNet-50 IR stream, fused through gated cross-modal modules. IC-Fusion reports 70.3 mAP on LLVIP and 46.1 mAP on FLIR-aligned, matching or beating symmetric ResNet-50/ResNet-50 fusion while using fewer parameters and multiply-accumulate operations. A reader should care because the claim, if true, redirects multispectral detector design toward asymmetric, infrared-prioritized architectures rather than equal-weight fusion.

What carries the argument

The central object is IC-Fusion, an infrared-centric multispectral detection transformer built on the RT-DETR detector. Its load-bearing asymmetry is that the RGB stream uses a compact ResNet-18 backbone while the IR stream uses a deeper ResNet-50 backbone. The fusion machinery is a cascade of four modules: Multi-Scale Feature Distillation (MSFD), which enriches RGB features with multi-branch depthwise convolutions; Cross-Modal Channel Shuffle Gate (CCSG), which reorganizes channels across modalities and gates them with GELU; Cross-Modal Large Kernel Gate (CLKG), which applies large-kernel depthwise convolutions as a spatial gate to align long-range structure; and Channel Shuffle Projection (CSP), which distills the concatenated multi-scale features with grouped pointwise convolutions. Together they let the model emphasize informative cross-modal features and suppress redundant ones, enabling the asymmetric backbone split to work.

What would settle it

Run IC-Fusion against a symmetric ResNet-50/ResNet-50 fusion on a multispectral benchmark where visible images carry most of the boundary detail, such as a daytime high-texture scene set, or compute wavelet sub-band energies across FLIR and LLVIP and correlate them with per-image mAP; if the symmetric model wins or RGB-heavy high-frequency images do not favor the IR-centric design, the central claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that IR images are not merely a fallback for low-light conditions but the structurally dominant modality: their high-frequency sub-bands contain the boundaries and contours that localization depends on, while RGB mainly provides low-frequency semantic context. From this the paper derives a design principle, allocate more capacity to IR and keep the RGB branch deliberately lightweight, and instantiates it in IC-Fusion with ResNet-50 for IR, ResNet-18 for RGB, and a fusion module where MSFD enriches RGB features, CCSG and CLKG gate cross-modal features, and CSP projects the result. The reported outcome is a state-of-the-art 70.3 mAP on LLVIP and 46.1 mAP on FLIR-aligned, with an ablation showing that the asymmetric backbone choice beats the reversed assignment and the symmetric ResNet-50/ResNet-50 configuration while using fewer parameters.

Load-bearing premise

The load-bearing premise is that infrared images generally hold the high-frequency boundary structure that object detection needs, so the deep network should go to IR and the RGB stream can be lightweight; this is supported by one wavelet example and by unimodal FLIR results, but it is assumed to hold across other datasets and lighting conditions.

Editorial extensions

If this is right

  • If IR is structurally dominant, multispectral detectors can shift compute from RGB to IR without losing accuracy, as shown by the ResNet-18/ResNet-50 configuration beating ResNet-50/ResNet-50 at lower cost.
  • Each component in the fusion cascade contributes measurably on FLIR-aligned, so the full MSFD-CCSG-CLKG sequence, not any single gate, is what carries the gain.
  • Since unimodal IR already outperforms unimodal RGB on FLIR across backbone types, IR-centric fusion should also hold on other benchmarks where thermal contours dominate over visible texture.
  • Because the method is built on RT-DETR, the infrared-centric fusion principle can be transferred to other DETR-family detectors without redesigning the encoder-decoder.
  • The two-stage training strategy, pretraining on modality-isolated data before fusion training, is part of what makes the asymmetric design work on the limited aligned data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper, the wavelet evidence is a single illustrative decomposition rather than a statistical link between sub-band energy and detection accuracy; measuring that link across FLIR and LLVIP would tell whether the IR-centric premise is quantitative or qualitative.
  • If IR-centricity generalizes, the likely failure boundary is daytime, high-texture scenes where visible images carry most boundary detail; in those conditions adaptive routing or symmetric fusion could beat a fixed IR-prioritized design.
  • The lightweight gating primitives, channel shuffle plus GELU gating, could plausibly transfer to other cross-modal tasks such as semantic segmentation or visual-infrared tracking, not just object detection.
  • A testable extension is to apply the same asymmetric backbone split to other transformer detectors and see whether the LLVIP and FLIR margins persist independently of RT-DETR's specific encoder.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. IC-Fusion is a multispectral object detector built on the RT-DETR transformer framework with deliberately asymmetric backbones: a lightweight ResNet-18 for the RGB modality and a deeper ResNet-50 for the IR modality. The design is motivated by a wavelet decomposition (Fig. 1) and by unimodal FLIR results (Table I) suggesting that IR carries the high-frequency structural cues most critical for detection. A fusion module comprising Multi-Scale Feature Distillation (MSFD) for the RGB stream, a Cross-Modal Channel Shuffle Gate (CCSG), a Cross-Modal Large Kernel Gate (CLKG), and a Channel Shuffle Projection (CSP) is inserted before the transformer encoder. The paper reports FLIR-aligned mAP of 46.1 and LLVIP mAP of 70.3, claims state-of-the-art performance on LLVIP, and presents ablations on backbone asymmetry (Table III) and on each proposed component (Table IV), along with parameter/MAC comparisons (Table V).

Significance. The paper has several concrete strengths: it releases code, evaluates on the two standard benchmarks in the area (FLIR-aligned and LLVIP), and the component ablation in Table IV is directionally consistent, with each added module improving the previous configuration. The central design hypothesis — that a deliberately weakened RGB branch combined with a deep IR branch can match or exceed symmetric dual-ResNet-50 fusion — is falsifiable and worth testing. However, the significance is currently bounded by three factors. First, the reported advantages over GM-DETR are extremely thin (70.3 vs 70.2 mAP on LLVIP; 46.1 vs 45.8 mAP on FLIR) with no error bars or multiple-seed reporting. Second, the motivating premise that IR is the dominant modality is contradicted by the paper's own unimodal LLVIP numbers, where RGB RT-DETR achieves 67.9 mAP versus 54.2 mAP for IR. Third, the claim in Section III.B that the asymmetric setup 'even outperforms the dual ResNet-50 configuration' is not supported by Table III on the primary metric (46.2 vs 45.8 mAP).

major comments (4)
  1. [Tables II-IV, Section III.B] The same configuration — ResNet-18 RGB backbone, ResNet-50 IR backbone, and the full MSFD+CCSG+CLKG fusion module — is reported with different performance numbers on FLIR-aligned across tables. Table II lists mAP50 83.3 / mAP75 42.6 / mAP 46.1 for IC-Fusion, while Table III (ResNet18/ResNet50 row) and Table IV (final ablation row) list 83.9 / 42.6 / 45.8 for the same model. Because these tables describe the identical architecture, the 0.6-point mAP50 and 0.3-point mAP discrepancies must be explained (e.g., different seeds, checkpoints, or evaluation settings); without such an explanation the headline FLIR result and the ablation baseline are mutually inconsistent, and the reader cannot tell which numbers are the definitive ones.
  2. [Section III.B, Table III] The sentence 'our asymmetric setup even outperforms the dual ResNet-50 configuration despite having fewer parameters and reduced computational cost' is not supported by Table III on the paper's primary metric. Table III reports mAP 46.2 for ResNet50+ResNet50 versus 45.8 for ResNet18+ResNet50; the asymmetric configuration leads only on mAP50 (83.9 vs 83.4) and ties on mAP75 (42.6 vs 42.4). Since the experimental setup defines mAP averaged over IoU 0.50–0.95 as the headline measure, the symmetric dual-ResNet-50 setup is more accurate on the primary metric. The claim should be restated as an efficiency-accuracy trade-off, or the text revised, because as written the central comparative claim for the IR-centric asymmetric design is contradicted by the paper's own data.
  3. [Section I, Tables I-II] The motivating premise that IR images contain 'structurally rich high-frequency information critical for object detection' is supported only on FLIR-aligned and is contradicted on LLVIP, where RT-DETR in RGB mode reaches 67.9 mAP versus 54.2 mAP in IR mode (Table II). The wavelet evidence in Fig. 1 is a single qualitative example with no quantitative link between sub-band energy and detection accuracy. Because the architecture deliberately assigns the deeper backbone to IR and the shallower backbone to RGB, the paper should either restrict the motivating claim to FLIR-like conditions or provide a quantitative analysis (e.g., sub-band energy statistics across both datasets, or a backbone-depth-controlled experiment) explaining why an IR-centric design is justified on a benchmark where unimodal IR is substantially weaker.
  4. [Section III, Tables II-IV] No error bars, seeds, or number of runs are reported anywhere in the experiments, and the decisive margins are very small: 70.3 vs 70.2 mAP on LLVIP against GM-DETR, 46.1 vs 45.8 mAP on FLIR, and ablation increments of 0.4–0.7 mAP in Table IV. For a DETR-style training pipeline with 60 epochs and a batch size of 8, these differences are likely within run-to-run variance. The authors should report the mean and standard deviation over at least three seeds and state explicitly whether the state-of-the-art claim and the component-ablation claims survive that comparison.
minor comments (5)
  1. [Table I] The mAP75 entry for Yolov5 (IR) is a dash even though mAP75 is reported for all other rows in the table; either provide the value or mark the column as not evaluated for that model.
  2. [Reference [5]] The reference title contains a typo: 'Generalized muiltispectral' should read 'Generalized Multispectral'.
  3. [Section III.A] The two-stage training strategy is described in a single sentence; it would help to specify concretely what 'modality-isolated pretraining' involves (epochs, data splits, whether the fusion modules are trained in the second stage from scratch) and to state whether the same protocol is used for the ablations in Tables III and IV.
  4. [Section III.B, Table V] The efficiency comparison reports parameters and MACs but not wall-clock latency; given the paper's 'lightweight' and 'efficient' claims, an FPS measurement on a common GPU would strengthen the comparison, and the input resolution used for the MACs computation (presumably 640x640) should be stated.
  5. [Section III.B] The statement that IC-Fusion 'consistently outperforms prior fusion methods' is stronger than Table II supports, since GM-DETR still leads on FLIR mAP50 (83.9 vs 83.3) and ties on mAP75 (42.6 vs 42.6); the claim should be qualified to the metrics and datasets where the advantage actually holds.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains come from held-out benchmark evaluation, not from fitted inputs or self-citation.

full rationale

IC-Fusion is an empirical architecture paper. The central claims — that an infrared-centric fusion design with a ResNet-18 RGB backbone, a ResNet-50 IR backbone, and the proposed MSFD/CCSG/CLKG/CSP modules improves detection mAP on FLIR-aligned and LLVIP — are validated by training on the official train splits and reporting held-out test mAP, with ablations in Tables III and IV. No parameter is fitted to the test set and then renamed a prediction; the reported numbers are direct benchmark outcomes. The two-stage training recipe is borrowed from an external prior work (GM-DETR, ref [5]) and applied uniformly to all configurations, so it does not force the reported advantage. The wavelet motivation (Fig. 1) and unimodal FLIR results are empirical evidence for the design hypothesis, not a derivation whose conclusion is encoded in its premise. There are no self-citations by the present authors (Hwang, Han, Jeon), so no load-bearing self-citation chain exists. One inconsistency is noted for correctness rather than circularity: Section III.B claims the asymmetric setup 'even outperforms the dual ResNet-50 configuration,' whereas Table III reports dual ResNet-50 at mAP 46.2 vs 45.8 for ResNet18(RGB)+ResNet50(IR); this overstates the table but does not make any claimed result equivalent to its inputs. Overall, no circular step is present.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim depends on benchmark assumptions, training recipe choices, and a qualitative frequency-domain premise. Architectural hyperparameters are not derived and are therefore listed as free design choices.

free parameters (3)
  • Fusion kernel sizes = 3x3, 5x5, 5x5 dilated (rate 2)
    Hand-chosen for MSFD and CLKG; no sensitivity analysis reported. These are not fitted constants in a derivation, but they are free design choices that affect performance.
  • Backbone asymmetry = ResNet-18 (RGB), ResNet-50 (IR)
    Selected based on ablation Table III; it is an empirically validated design choice rather than a value derived from theory.
  • Training schedule hyperparameters = 60 epochs, lr 1e-4, backbone lr 1e-5, batch 8
    Adopted from RT-DETR/GM-DETR recipes; no systematic hyperparameter search or sensitivity analysis is reported.
assumptions (3)
  • domain assumption ImageNet-pretrained ResNet weights transfer to single-channel thermal IR input.
    Section III-A initializes backbones with ImageNet weights and trains on paired RGB/IR data; no analysis of domain shift is provided.
  • domain assumption The FLIR-aligned split curated by [8] is an appropriate and fair evaluation benchmark.
    All comparisons in Table II use this split; differences from the raw FLIR dataset could affect generalization claims.
  • ad hoc to paper High-frequency sub-bands of IR are the critical cue for detection.
    Section I supports this with one qualitative wavelet figure and unimodal detector scores; no quantitative ablation of frequency bands is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multispectral Detection Transformer with Infrared-Centric Feature Fusion." pith.science (2026). https://pith.science/paper/WEUEWHMO

@misc{pith2026250515137,
  author       = {Pith},
  title        = {Pith review of: Multispectral Detection Transformer with Infrared-Centric Feature Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WEUEWHMO}},
  note         = {Machine review of arXiv:2505.15137}
}
read the original abstract

Multispectral object detection aims to leverage complementary information from visible (RGB) and infrared (IR) modalities to enable robust performance under diverse environmental conditions. Our key insight, derived from wavelet analysis and empirical observations, is that IR images contain structurally rich high-frequency information critical for object detection, making an infrared-centric approach highly effective. To capitalize on this finding, we propose Infrared-Centric Fusion (IC-Fusion), a lightweight and modality-aware sensor fusion method that prioritizes infrared features while effectively integrating complementary RGB semantic context. IC-Fusion adopts a compact RGB backbone and designs a novel fusion module comprising a Multi-Scale Feature Distillation (MSFD) block to enhance RGB features and a three-stage fusion block with a Cross-Modal Channel Shuffle Gate (CCSG), a Cross-Modal Large Kernel Gate (CLKG), and a Channel Shuffle Projection (CSP) to facilitate effective cross-modal interaction. Experiments on the FLIR and LLVIP benchmarks demonstrate the superior effectiveness and efficiency of our IR-centric fusion strategy, further validating its benefits. Our code is available at https://github.com/smin-hwang/IC-Fusion.

Figures

Figures reproduced from arXiv: 2505.15137 by the authors.

Figure 1
Figure 1. Wavelet decomposition of visible (RGB) and infrared (IR) images. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of (a) the proposed IC-Fusion framework and (b) the [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. Qualitative results on FLIR using RGB, IR, and our fusion model. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 14 canonical work pages

  1. [1]

    Guided attentive feature fusion for multispectral pedestrian detection,

    H. Zhang, E. Fromont, S. Lef `evre, and B. Avignon, “Guided attentive feature fusion for multispectral pedestrian detection,” in Proceedings of the IEEE/CVF winter conference on applications of computer vision , 2021, pp. 72–80

    work page 2021

  2. [2]

    Stabilizing multispectral pedestrian detection with evidential hybrid fusion,

    Q. Li, C. Zhang, Q. Hu, P. Zhu, H. Fu, and L. Chen, “Stabilizing multispectral pedestrian detection with evidential hybrid fusion,” IEEE Transactions on Circuits and Systems for Video Technology , vol. 34, no. 4, pp. 3017–3029, 2023

    work page 2023

  3. [3]

    Multi-modal feature pyramid transformer for rgb-infrared object detection,

    Y . Zhu, X. Sun, M. Wang, and H. Huang, “Multi-modal feature pyramid transformer for rgb-infrared object detection,” IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 9, pp. 9984–9995, 2023

    work page 2023

  4. [4]

    MS-DETR: Multispectral Pedestrian Detection Transformer with Loosely Coupled Fusion and Modality-Balanced Optimization

    Y . Xing, S. Wang, G. Liang, Q. Li, X. Zhang, S. Zhang, and Y . Zhang, “Multispectral pedestrian detection via reference box con- strained cross attention and modality balanced optimization,” arXiv preprint arXiv:2302.00290, vol. 1, no. 2, p. 5, 2023

    work page Pith review arXiv 2023

  5. [5]

    Gm-detr: Generalized muiltispectral detection transformer with efficient fusion encoder for visible-infrared detection,

    Y . Xiao, F. Meng, Q. Wu, L. Xu, M. He, and H. Li, “Gm-detr: Generalized muiltispectral detection transformer with efficient fusion encoder for visible-infrared detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 5541–5549

    work page 2024

  6. [6]

    Locnet: Improving localization accu- racy for object detection,

    S. Gidaris and N. Komodakis, “Locnet: Improving localization accu- racy for object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 789–798

    work page 2016

  7. [7]

    segdeepm: Exploiting segmentation and context in deep neural networks for object detection,

    Y . Zhu, R. Urtasun, R. Salakhutdinov, and S. Fidler, “segdeepm: Exploiting segmentation and context in deep neural networks for object detection,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 4703–4711

    work page 2015

  8. [8]

    Multispectral fusion for object detection with cyclic fuse-and-refine blocks,

    H. Zhang, E. Fromont, S. Lefevre, and B. Avignon, “Multispectral fusion for object detection with cyclic fuse-and-refine blocks,” in 2020 IEEE International conference on image processing (ICIP) . IEEE, 2020, pp. 276–280

    2020

Show all 23 references
  1. [9]

    Detrs beat yolos on real-time object detection,

    Y . Zhao, W. Lv, S. Xu, J. Wei, G. Wang, Q. Dang, Y . Liu, and J. Chen, “Detrs beat yolos on real-time object detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 16 965–16 974

    2024

  2. [10]

    Faster r-cnn: Towards real-time object detection with region proposal networks,

    S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” Advances in neural information processing systems , vol. 28, 2015

    2015

  3. [11]

    Cross-modality fu- sion transformer for multispectral object detection,

    F. Qingyun, H. Dapeng, and W. Zhaokui, “Cross-modality fu- sion transformer for multispectral object detection,” arXiv preprint arXiv:2111.00273, 2021

    2021 arXiv

  4. [12]

    End-to-end object detection with transformers,

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in European conference on computer vision . Springer, 2020, pp. 213– 229

    2020

  5. [13]

    Deformable detr: De- formable transformers for end-to-end object detection,

    X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: De- formable transformers for end-to-end object detection,” in International Conference on Learning Representations , 2020

    2020

  6. [14]

    Dino: Detr with improved denoising anchor boxes for end-to-end object detection,

    H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. Ni, and H.-Y . Shum, “Dino: Detr with improved denoising anchor boxes for end-to-end object detection,” in International Conference on Learning Representations , 2020

    2020

  7. [15]

    Multimodal object detection by channel switching and spatial attention,

    Y . Cao, J. Bin, J. Hamari, E. Blasch, and Z. Liu, “Multimodal object detection by channel switching and spatial attention,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 403–411

    2023

  8. [16]

    Lraf-net: Long-range attention fusion network for visible–infrared object detec- tion,

    H. Fu, S. Wang, P. Duan, C. Xiao, R. Dian, S. Li, and Z. Li, “Lraf-net: Long-range attention fusion network for visible–infrared object detec- tion,” IEEE Transactions on Neural Networks and Learning Systems , 2023

    2023

  9. [17]

    Icafusion: Iterative cross-attention guided feature fusion for multispectral object detection,

    J. Shen, Y . Chen, Y . Liu, X. Zuo, H. Fan, and W. Yang, “Icafusion: Iterative cross-attention guided feature fusion for multispectral object detection,” Pattern Recognition, vol. 145, p. 109913, 2024

    2024

  10. [18]

    Gaussian error linear units (gelus),

    D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  11. [19]

    Language modeling with gated convolutional networks,

    Y . N. Dauphin, A. Fan, M. Auli, and D. Grangier, “Language modeling with gated convolutional networks,” in International conference on machine learning. PMLR, 2017, pp. 933–941

    2017

  12. [20]

    Free teledyne flir thermal dataset for algorithm training,

    “Free teledyne flir thermal dataset for algorithm training,” https://www. flir.in/oem/adas/adas-dataset-form, 2018, accessed: 2025-03-23

    2018

  13. [21]

    Llvip: A visible-infrared paired dataset for low-light vision,

    X. Jia, C. Zhu, M. Li, W. Tang, and W. Zhou, “Llvip: A visible-infrared paired dataset for low-light vision,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 3496–3504

    2021

  14. [22]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778

    2016

  15. [23]

    Meta, “fvcore,” https://github.com/facebookresearch/fvcore, 2023, ac- cessed: 2025-03-24

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.