REVIEW 4 major objections 5 minor 26 references
End-to-End RGB-IR Joint Image Compression With Channel-wise Cross-modality Entropy Model
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that joint RGB-infrared compression beats separate and sequential codecs by using low-frequency context from the infrared latent to condition the RGB entropy model, achieving a 23.1% bit-rate saving over the previous…
desk verdict A plausible architecture undermined by an input-format asymmetry that likely explains a large part of the reported BD-rate gains. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Channel-wise Cross-modality Entropy Model (CCEM). It takes the quantized latents of the two modalities, splits each into five slices, and uses a hyperprior plus already-coded slices to predict each slice's distribution parameters. Two new blocks carry the cross-modality work: the Low-frequency Context Extraction Block (LCEB) uses a global-attention transformer to pull global low-frequency context from the decoded slices of a modality, and the Low-frequency Context Fusion Block (LCFB) uses agent attention to selectively aggregate the low-frequency context of the IR latent into the RGB latent being coded. The fused low-frequency context is what allows the RGB entropy model to exploit cross-modality redundancy; the IR latent is coded first with only intra-modality and hyperprior context.
What would settle it
Compress a set of RGB-IR pairs that are deliberately misaligned or where the infrared channel contains content with no RGB counterpart (e.g., a heat source hidden behind an object). If the bit-rate savings over single-modality coding persist on such pairs, the low-frequency-similarity premise is not the operative cause; if the savings collapse, the premise is confirmed as the load-bearing assumption.
Extended reading notes
Core claim
The paper's central claim is that in RGB-IR pairs from the same scene, the two modalities share global low-frequency content, and this shared content can be extracted from the infrared latent and used as a conditioning prior for the RGB entropy model. Building on channel-wise autoregressive entropy models, the authors split each latent into slices, code the IR latent first, and for each RGB slice fuse the decoded IR slices' low-frequency context with the RGB slice's own low-frequency context via an agent-attention fusion block; the fused context is concatenated with hyperprior and slice context to predict entropy parameters. The result, they report, is a 23.1% average bit-rate saving over the CVPR2022 RGB-IR codec and 14.6% over MLIC++ on LLVIP, with the IR modality alone gaining 35.1% BD-rate over BPG, while visual inspection shows the method preserves small text and plate numbers at lower bit rates. The paper also reports that the asymmetric direction (IR context assisting RGB) beats the reverse direction by about 8% bit rate, which they attribute to IR features carrying noise when injected into the RGB model.
Load-bearing premise
The method assumes that global low-frequency information in the RGB and infrared views of the same scene is highly similar, so that low-frequency context extracted from the infrared latent can accurately predict RGB entropy parameters; if that similarity is weak for the target data, the cross-modality prior would not produce the reported gains.
Editorial extensions
If this is right
- RGB-IR pairs can be compressed in one joint pass, removing the latency of anchor-based schemes that must decode one modality before coding the other.
- Using low-frequency context rather than full feature maps as the cross-modal prior keeps the conditioning overhead small while delivering the reported bit-rate gains.
- Both modalities get rate-distortion improvements, with the IR modality gaining more (35.1% BD-rate saving over BPG on LLVIP) than the RGB modality (19.9%).
- The two-stage training scheme, where the model first learns RGB compression and then is fine-tuned jointly, adds about 4% BD-rate improvement over joint training from scratch.
Reading between the lines
- If the low-frequency-sharing premise generalizes, the same conditioning pattern should transfer to other paired sensor modalities whose low-frequency content is strongly correlated, such as RGB and depth or multispectral bands, where the expected savings may be larger.
- The measured asymmetry (IR assisting RGB helps more than the reverse) suggests a design principle for cross-modal codecs: condition the modality with richer texture on the modality with cleaner low-frequency structure, rather than broadcasting the noisier modality's details into the cleaner one.
- A straightforward stress test would be to run the same architecture on deliberately misaligned RGB-IR pairs or on scenes where the IR view contains content with no RGB counterpart; the reported savings should concentrate on pairs with high low-frequency cross-correlation.
- Because the target applications are surveillance and driving, the framework could be evaluated with downstream task metrics such as pedestrian detection accuracy instead of only PSNR/MS-SSIM, quantifying whether the retained information is the task-relevant information.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a dual-branch learned compression framework for aligned RGB-IR image pairs. The RGB image is first converted to YUV420, and the Y, U, V, and IR channels are encoded by a transformer-based autoencoder; the latent YUV and IR representations are jointly entropy-coded by a Channel-wise Cross-modality Entropy Model (CCEM). The CCEM uses a hyperprior together with Low-frequency Context Extraction Blocks (LCEB) and Low-frequency Context Fusion Blocks (LCFB) to predict entropy parameters from within-modality and cross-modality low-frequency context. The model is trained on LLVIP and KAIST and evaluated against CVPR2022 RGB-IR compression, MLIC++, Cheng2020, and BPG, reporting a 23.1% BD-rate saving over CVPR2022 and a 14.6% saving over MLIC++ on LLVIP, plus per-modality BD-rate tables, runtime and memory measurements, and an ablation study.
Significance. If the reported gains hold under matched evaluation conditions, the paper makes a useful contribution to multi-modal learned image compression: it demonstrates that cross-modality low-frequency priors can improve entropy estimation for RGB-IR pairs and that joint compression can outperform separate single-modality coding. The paper includes comparisons on two multi-modal datasets, per-modality BD-rate numbers, and explicit runtime/memory reporting, which are valuable for reproducibility. Its central claim, however, rests on a fair comparison between the proposed architecture and baselines, and the current experimental protocol does not establish that fairness. The architecture description also omits at least one visible component (LRP in Fig. 3), and the ablation table is difficult to interpret. These issues are fixable but are load-bearing for the headline results.
major comments (4)
- [Sec. II-A / Sec. III-A] The reported BD-rate gains may be an input-format artifact. Section II-A states that before compression the RGB image is converted to YUV420, so the proposed method codes quarter-resolution U and V channels, whereas the RGB baselines (Cheng2020, MLIC++) are standard three-channel codecs and the paper does not state that they were given subsampled YUV420 input. Section III-A says 'all evaluation metrics are computed in the YUV420 domain,' which means a full-resolution RGB baseline spends rate on chroma information that is discarded or downsampled during evaluation. This asymmetry is independent of the proposed entropy model and can inflate the apparent savings. Please rerun the comparisons with matched input formats: either encode all RGB baselines from YUV420 inputs using the same preprocessing, or encode the proposed method from full-resolution RGB, and report BD-rate under both conditions.
- [Sec. III-A] The IR baseline comparison is also asymmetric. Section III-A says that for single-modality codecs the single-channel IR images are duplicated into three channels to fit the pre-trained model structure, while the proposed method codes IR through a dedicated one-channel branch. This forces MLIC++ and Cheng2020 to spend rate on two redundant duplicate channels, handicapping them independently of the CCEM. The BPG grayscale option is not affected, but the learned baselines are. Please adapt the learned single-modality baselines to accept one-channel IR input (e.g., by changing the first convolution layer and fine-tuning), or alternatively report the proposed method with IR duplicated into three channels, so that the comparison isolates the architecture rather than the input format.
- [Table II] The ablation study as presented does not support the attribution of gains to LCEB and LCFB. The rows 'baseline -', 'Channel-wise Cross-modality Entropy Model -19.34', 'baseline + LCEB -6.92', and 'baseline + LCFB -9.17' are internally inconsistent: it is unclear what 'baseline' denotes, why the full CCEM row appears before the additive rows, and what the baseline BD-rate value actually is. Without a clearly defined baseline and without reporting the BD-rate of the full model with each module removed individually, the reader cannot tell whether the two blocks contribute positively or whether the gains come from the overall CCEM design. Please provide a complete ablation table with explicit model definitions, the baseline BD-rate value, and matched input formats for every row.
- [Fig. 3 / Sec. II-B] Figure 3 shows a 'LRP' module labeled 'latent residual prediction module,' but the text never describes this component or its role in the entropy model. If LRP is part of the proposed method, omitting its description prevents reproduction and makes it impossible to determine whether the reported gains are due to the cross-modality low-frequency context or to this unexplained module. Please either add a full description of LRP and its training/inference behavior, or remove it from the architecture figure if it is not used in the final model.
minor comments (5)
- [Sec. III-B] The headline BD-rate numbers (23.1% vs CVPR2022 and 14.6% vs MLIC++) are quoted in the text but no table reports the average-PSNR BD-rate matrix across all methods and datasets; Table I only gives per-modality BD-rate against BPG. Please include the full average BD-rate table with all pairs of methods.
- [Sec. III-A] No error bars, multiple-seed runs, or statistical significance tests are reported for any BD-rate or PSNR figure. Given that the reported gains are on the order of a few percent and the comparison protocol is currently unmatched, single-run numbers are insufficient. Please report at least three seeds or a confidence interval for the main comparisons.
- [Sec. II-B] The text says 'we adopt the Lite Transformer architecture[21],' but reference [21] is CDDFuse, not the Lite Transformer paper; please correct the citation. There are also minor typographical issues such as 'to exact context' (should be 'to extract context') and 'Ffusionas' (missing space).
- [Sec. III-B / Table I] The per-modality BD-rate values in Table I use inconsistent decimal precision (e.g., 5.426 vs -31.052? and -19.89 vs -35.051); please standardize the number of decimals so the table is readable and unambiguous.
- [Sec. II-B / Sec. III-B] The paper does not discuss how the RGB and IR images are spatially aligned or registered in the LLVIP and KAIST datasets. The cross-modality low-frequency prior assumes the two modalities are co-registered; a sentence on alignment or its absence would help the reader assess the generality of the method.
Circularity Check
No circular derivation: the claimed BD-rate gains come from measured comparisons against external baselines, not from fitting or self-citation; the YUV420-input evaluation asymmetry is a benchmarking concern, not a circularity.
full rationale
The paper's central claim is empirical: a jointly trained RGB-IR compression model with a Channel-wise Cross-modality Entropy Model (CCEM) is measured against external baselines (CVPR2022, MLIC++, Cheng2020, BPG) on LLVIP and KAIST. The design premise that global low-frequency information of RGB and IR images from the same scene is highly similar is imported from the external reference [21] (CDDFuse), and the hyperprior structure follows the external method [12]; neither is a self-citation chain, and neither is a uniqueness theorem invoked to forbid alternatives. The probability factorizations in Eq. (3) are standard conditional entropy-model definitions and do not by construction imply the 23.1% BD-rate saving. No parameter is fitted to the test-set BD-rate and then renamed a prediction; the ablation study, while incomplete, compares component variants and reports measured differences. The paper does state that 'Before compression, the RGB image is converted to the YUV420 format' and that 'all evaluation metrics are computed in the YUV420 domain,' which raises a legitimate concern that the comparison may be unfair to RGB baselines that encode full-resolution chroma, and IR baselines are handicapped by 3-channel duplication. However, this is an experimental-validity threat, not a circular reduction in the derivation chain, and therefore does not raise the circularity score. No circular step can be exhibited from the quoted equations or citation structure.
Assumptions & free parameters
free parameters (3)
- rate-distortion tradeoff lambda =
[0.0018, 0.0035, 0.0067, 0.0130, 0.0250, 0.0483]
- number of latent slices N =
5
- latent representation channel count =
320
assumptions (3)
- domain assumption Global low-frequency information of RGB and IR images from the same scene is highly similar.
- domain assumption Lite Transformer's long-range modeling is suitable for capturing low-frequency context.
- ad hoc to paper Agent-attention yields more effective cross-modal fusion than concatenation.
Cite this review
Pith. "Pith review of End-to-End RGB-IR Joint Image Compression With Channel-wise Cross-modality Entropy Model." pith.science (2026). https://pith.science/paper/45JYLUHN
@misc{pith2026250621851,
author = {Pith},
title = {Pith review of: End-to-End RGB-IR Joint Image Compression With Channel-wise Cross-modality Entropy Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/45JYLUHN}},
note = {Machine review of arXiv:2506.21851}
}
read the original abstract
RGB-IR(RGB-Infrared) image pairs are frequently applied simultaneously in various applications like intelligent surveillance. However, as the number of modalities increases, the required data storage and transmission costs also double. Therefore, efficient RGB-IR data compression is essential. This work proposes a joint compression framework for RGB-IR image pair. Specifically, to fully utilize cross-modality prior information for accurate context probability modeling within and between modalities, we propose a Channel-wise Cross-modality Entropy Model (CCEM). Among CCEM, a Low-frequency Context Extraction Block (LCEB) and a Low-frequency Context Fusion Block (LCFB) are designed for extracting and aggregating the global low-frequency information from both modalities, which assist the model in predicting entropy parameters more accurately. Experimental results demonstrate that our approach outperforms existing RGB-IR image pair and single-modality compression methods on LLVIP and KAIST datasets. For instance, the proposed framework achieves a 23.1% bit rate saving on LLVIP dataset compared to the state-of-the-art RGB-IR image codec presented at CVPR 2022.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Cross-modality paired-images generation for rgb-infrared person re-identification,
G.-A. Wang, T. Zhang, Y . Yang, J. Cheng, J. Chang, X. Liang, and Z.-G. Hou, “Cross-modality paired-images generation for rgb-infrared person re-identification,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 07, 2020, pp. 12 144–12 151
work page 2020
-
[2]
Tfdet: Target-aware fusion for rgb-t pedestrian detection,
X. Zhang, X. Zhang, J. Wang, J. Ying, Z. Sheng, H. Yu, C. Li, and H.- L. Shen, “Tfdet: Target-aware fusion for rgb-t pedestrian detection,” IEEE Transactions on Neural Networks and Learning Systems , 2024
work page 2024
-
[3]
S. Lee, T. Kim, J. Shin, N. Kim, and Y . Choi, “Insanet: Intra-inter spectral attention network for effective feature fusion of multispectral pedestrian detection,” Sensors, vol. 24, no. 4, p. 1168, 2024
work page 2024
-
[4]
Rgb- infrared cross-modality person re-identification via joint pixel and feature alignment,
G. Wang, T. Zhang, J. Cheng, S. Liu, Y . Yang, and Z. Hou, “Rgb- infrared cross-modality person re-identification via joint pixel and feature alignment,” in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 3623–3632
work page 2019
-
[5]
End-to-end optimized image compression,
J. Ball ´e, V . Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” arXiv preprint arXiv:1611.01704 , 2016
arXiv 2016
-
[6]
Learned image com- pression with discretized gaussian mixture likelihoods and attention modules,
Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learned image com- pression with discretized gaussian mixture likelihoods and attention modules,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 7939–7948
work page 2020
-
[7]
Coarse-to-fine hyper-prior modeling for learned image compression,
Y . Hu, W. Yang, and J. Liu, “Coarse-to-fine hyper-prior modeling for learned image compression,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 34, no. 07, 2020, pp. 11 013–11 020
work page 2020
-
[8]
Transformer-based transform coding,
Y . Zhu, Y . Yang, and T. Cohen, “Transformer-based transform coding,” in International Conference on Learning Representations , 2022
work page 2022
Show all 26 references
-
[9]
Elic: Efficient learned image compression with unevenly grouped space- channel contextual adaptive coding,
D. He, Z. Yang, W. Peng, R. Ma, H. Qin, and Y . Wang, “Elic: Efficient learned image compression with unevenly grouped space- channel contextual adaptive coding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 5718–5727
2022
-
[10]
Mlic: Multi-reference entropy model for learned image compression,
W. Jiang, J. Yang, Y . Zhai, P. Ning, F. Gao, and R. Wang, “Mlic: Multi-reference entropy model for learned image compression,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 7618–7627
2023
-
[11]
Mixture autoregressive and spectral attention network for multispectral image compression based on variational autoencoder,
F. Kong, G. Ren, Y . Hu, D. Li, and K. Hu, “Mixture autoregressive and spectral attention network for multispectral image compression based on variational autoencoder,” The Visual Computer, vol. 40, no. 9, pp. 6295–6318, 2024
2024
-
[12]
End-to-end rgb-d image compression via exploiting channel-modality redundancy,
H. Zheng and W. Gao, “End-to-end rgb-d image compression via exploiting channel-modality redundancy,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 7, 2024, pp. 7562– 7570
2024
-
[13]
Learning based multi-modality image and video compression,
G. Lu, T. Zhong, J. Geng, Q. Hu, and D. Xu, “Learning based multi-modality image and video compression,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 6083–6092
2022
-
[14]
Multispec- tral deep neural networks for pedestrian detection,
J. Liu, S. Zhang, S. Wang, and D. N. Metaxas, “Multispec- tral deep neural networks for pedestrian detection,” arXiv preprint arXiv:1611.02644, 2016
2016 arXiv
-
[15]
Removal and selection: Improving rgb-infrared object detection via coarse-to-fine fusion,
T. Zhao, M. Yuan, F. Jiang, N. Wang, and X. Wei, “Removal and selection: Improving rgb-infrared object detection via coarse-to-fine fusion,” arXiv preprint arXiv:2401.10731 , 2024
2024
-
[16]
Llvip: A visible-infrared paired dataset for low-light vision,
X. Jia, C. Zhu, M. Li, W. Tang, and W. Zhou, “Llvip: A visible-infrared paired dataset for low-light vision,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 3496–3504
2021
-
[17]
Kaist multi-spectral day/night data set for autonomous and assisted driving,
Y . Choi, N. Kim, S. Hwang, K. Park, J. S. Yoon, K. An, and I. S. Kweon, “Kaist multi-spectral day/night data set for autonomous and assisted driving,” IEEE Transactions on Intelligent Transportation Systems, vol. 19, no. 3, pp. 934–948, 2018
2018
-
[18]
Swin transformer: Hierarchical vision transformer using shifted windows,
Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 10 012–10 022
2021
-
[19]
Joint autoregressive and hierarchical priors for learned image compression,
D. Minnen, J. Ball ´e, and G. D. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” Advances in neural information processing systems , vol. 31, 2018
2018
-
[20]
Channel-wise autoregressive entropy models for learned image compression,
D. Minnen and S. Singh, “Channel-wise autoregressive entropy models for learned image compression,” in 2020 IEEE International Confer- ence on Image Processing (ICIP) . IEEE, 2020, pp. 3339–3343
2020
-
[21]
Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,
Z. Zhao, H. Bai, J. Zhang, Y . Zhang, S. Xu, Z. Lin, R. Timofte, and L. Van Gool, “Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 5906–5916
2023
-
[22]
Agent attention: On the integration of softmax and linear attention,
D. Han, T. Ye, Y . Han, Z. Xia, S. Pan, P. Wan, S. Song, and G. Huang, “Agent attention: On the integration of softmax and linear attention,” in European Conference on Computer Vision . Springer, 2025, pp. 124–140
2025
-
[23]
Flatten transformer: Vision transformer using focused linear attention,
D. Han, X. Pan, Y . Han, S. Song, and G. Huang, “Flatten transformer: Vision transformer using focused linear attention,” in Proceedings of the IEEE/CVF international conference on computer vision , 2023, pp. 5961–5971
2023
-
[24]
Bpg image format,
F. Bellard, “Bpg image format,” Available: http://bellard.org/bpg/, 2018, accessed: Oct. 30, 2018
2018
-
[25]
Calculation of average psnr differences between rd- curves,
G. Bjontegaard, “Calculation of average psnr differences between rd- curves,” ITU-T, Tech. Rep. VCEG-M33, 2001
2001
-
[26]
Compressai: a pytorch library and evaluation platform for end-to-end compression research,
J. B ´egaint, F. Racap ´e, S. Feltman, and A. Pushparaja, “Compressai: a pytorch library and evaluation platform for end-to-end compression research,” arXiv preprint arXiv:2011.03029 , 2020
2011 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.