Pith. sign in

REVIEW 4 major objections 5 minor 87 references

ModalFormer: Multimodal Transformer for Low-Light Image Enhancement

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fusing nine auxiliary visual modalities into transformer attention improves low-light image enhancement beyond prior methods on all five benchmark datasets while using about 45% of the strongest baseline's parameters.

desk verdict Solid multimodal LLIE paper with real enhancement gains, but the computational-efficiency claim overreaches until the 4M-21 extractor is accounted for. read the letter →

arxiv 2507.20388 v1 pith:XVH3D7OK submitted 2025-07-27 cs.CV

Alexandru Brateanu , Raul Balmez , Ciprian Orhei , Codruta Ancuti , Cosmin Ancuti This is my paper · ORCID
classification cs.CV
keywords low-lightimageenhancementmultimodaltransformercross-modalattentionrestorationauxiliarymodalitiesfeaturereconstruction4M-21
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Low-light image enhancement is usually treated as a pixel-to-pixel RGB problem, and this paper argues that treating it as a multimodal problem works better. ModalFormer couples a U-shaped transformer with nine auxiliary modality streams—deep feature embeddings, segmentations, depth, surface normals, color palette, and luminance—extracted from the input by a frozen multimodal model. A new cross-modal attention module fuses these streams into hybrid attention maps, and auxiliary subnetworks reconstruct the modality maps under a dedicated loss. On the LOL-v1, LOL-v2 (real and synthetic), and SDSD (indoor and outdoor) benchmarks, the paper reports higher PSNR and SSIM than every compared method, including GLARE, while using 45% of GLARE's parameters. The ablations trace most of the gain to the cross-modal attention fusion and the multimodal reconstruction loss.

What carries the argument

The load-bearing component is the Cross-modal Multi-headed Self-Attention mechanism (CM-MSA) inside the Multimodal Transformer Block (MMTB). CM-MSA computes one attention map from the RGB tokens and nine separate attention maps from the injected auxiliary features at the same encoder–decoder levels; the nine maps are multiplied with learnable per-modality weights, softmaxed again, and applied to the RGB-attended values. This is what turns the transformer into a multimodal fuser rather than a single-stream restorer. Around it, the paper builds a U-shaped encoder–decoder (CM-T) and nine lightweight convolutional U-Nets that reconstruct the modality maps and feed them into the MMTBs; the auxiliary reconstruction loss L_MM is what keeps those maps informative during training.

What would settle it

Replace the nine extracted modality maps in ModalFormer with random tensors of identical shape during training, keeping the same loss and architecture; if LOL-v1 PSNR stays near the reported 27.97 dB, the improvement comes from extra parameters or training dynamics rather than from the multimodal information.

Watch

Extended reading notes

Core claim

The central claim is that injecting nine auxiliary modalities into a transformer's self-attention—rather than using RGB alone or concatenating features at the input—produces consistently better low-light enhancement. The paper states that ModalFormer surpasses previous methods on all five paired datasets, improving PSNR over GLARE by 0.62 dB on LOL-v1, 0.35 dB on LOL-v2-real, 0.31 dB on LOL-v2-synthetic, and 1.27 dB and 0.88 dB on SDSD indoor and outdoor, with SSIM gains on all but one. It further reports the best mean NIQE score of 3.61 on no-reference datasets and shows in ablations that replacing plain multi-headed self-attention with the cross-modal version adds 1.24 dB on LOL-v1, while the auxiliary losses contribute a further 0.45 dB. The authors offer this as evidence that rich contextual modalities, supervised by reconstructing the same modality maps from normal-light references, are what drive the improvement.

Load-bearing premise

The framework's gain rests on the assumption that the nine auxiliary maps pulled from a dim image are informative and complementary, and that the same extractor applied to a normal-light reference yields valid training targets for those maps.

Editorial extensions

If this is right

  • If the reported numbers hold, low-light enhancement no longer needs to trade quality for parameter count: ModalFormer beats the previous best with 19.81M parameters versus GLARE's 44.04M.
  • The cross-modal attention mechanism can be lifted out of this architecture and applied to other restoration tasks—denoising, deblurring, dehazing—where depth, segmentation, or color priors are available.
  • Because the auxiliary features are supervised against the same frozen extractor run on normal-light reference images, the training recipe offers a general way to define multimodal targets for paired restoration datasets.
  • The per-modality attention maps make the model's reasoning more inspectable: ablations already show feature embeddings and geometric cues carry the largest share of the gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the method's practical cost is dominated by the frozen modality extractor—the paper counts 295 ms of 728 ms on an RTX 3090—so a distilled or lighter extractor would likely preserve most of the quality gain while roughly halving runtime.
  • Editorial inference: nothing in the architecture forces all nine modalities; one could greedily prune modalities by zeroing their attention weights and measure PSNR, potentially reducing the system to the feature embeddings and geometry streams that the ablations already single out.
  • Editorial inference: the auxiliary ground-truth features come from normal-light images in paired datasets, so the recipe may transfer poorly to unpaired or misaligned capture conditions; a test is to train with one modality's targets corrupted by noise and see how gracefully the attention weights adapt.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents ModalFormer, a transformer-based low-light image enhancement method that fuses nine auxiliary modalities with RGB features. The architecture couples a U-shaped Cross-modal Transformer (CM-T) with nine convolutional U-Net auxiliary subnetworks. The core attention mechanism, CM-MSA, computes separate attention maps for RGB and each modality, multiplies them with learned balancing weights, and applies the fused map to the RGB values. Training uses a hybrid loss combining MSE, MS-SSIM, perceptual loss, and a multimodal reconstruction loss. The authors report state-of-the-art PSNR/SSIM on five benchmark datasets (LOL-v1, LOL-v2 real/synthetic, SDSD indoor/outdoor) and the best mean NIQE on four no-reference datasets, with code and models released.

Significance. If the reported improvements reproduce, ModalFormer is a meaningful step for LLIE, showing that large-scale multimodal features can be injected into a transformer restoration backbone with consistent PSNR gains over GLARE (0.31-1.27 dB across the five datasets) and better or comparable SSIM in four of five cases. The paper ships code and checkpoints, which supports verification. The main caveats are that the claimed computational advantage does not hold under the numbers the paper itself reports, and the 'surpasses all methods' phrasing is metric-dependent.

major comments (4)
  1. [4.2, Complexity Discussion] The statement in Section 4.2 that ModalFormer 'surpasses previous methods on all five datasets while reducing computational costs' is not supported on the cost side: the reported serial end-to-end latency is 728 ms (395 ms for ModalFormer plus 295 ms for 4M-21 extraction), which is slower than the quoted 650 ms for GLARE, and the 19.81M parameter count excludes the frozen 4M-21 extractor that is mandatory at inference. The 443 ms figure relies on parallel multimodal extraction via 4M-21, an assumption that is not stated in the abstract or in the Section 4.2 claim. Please rephrase the computational claim to specify 'trainable parameters' and 'with parallel 4M-21 extraction', or report a single-GPU end-to-end comparison.
  2. [4.2, Quantitative Results] The blanket claim 'surpasses previous methods on all five datasets' is false when all reported metrics are considered: Table 1 shows ModalFormer's SSIM on LOL-v2-synthetic is 0.951, below GLARE's 0.958, and Table 2 shows a higher NIQE on DICM (3.64 versus GLARE's 3.61). Please qualify the claim to state the metric (PSNR and mean NIQE) for which superiority holds.
  3. [3.1 and Eq. (5)] The auxiliary supervision L_MM and the injection of M1,...,M8 depend on a fixed extractor (4M-21) applied to both the low-light input and the normal-light reference, but the paper does not state whether the extractor is frozen during training, nor how the 22-channel features are obtained for non-dense modalities such as CLIP and ImageBind. This is central to the method's reproducibility; please specify the extraction protocol, the PCA details, and the training-time treatment of the extractor.
  4. [3.2.1, Eq. (3)] In Eq. (3), the product of nine attention matrices AMj is normalized by a softmax; however, the notation 'Q represents the matrix product' is confusing because Q already denotes the RGB query. Also, please specify how the k attention heads of CM-MSA interact with the single fused attention map ACM, since the current text defines a single map per stream but claims multi-head self-attention.
minor comments (5)
  1. [5, Table 3a] The plain MSA baseline (26.73 dB) referred to in the text is not shown in the table; please add it for transparency.
  2. [5, Modality Groups] The typo 'respectivelly' should be corrected to 'respectively', and the checkmark columns in Table 3b should be labeled explicitly so that the reader can see which modality is removed in each row.
  3. [3.2.1, FFN] The expansion factor mu is introduced as 'a hyperparameter set to 4' but is never named; please define it as the FFN expansion ratio before Eq. (4).
  4. [4.1, Datasets] The notation 'LOL-v2 (synthetic and real)' is later abbreviated inconsistently as 'LOL-v2-syn' and 'LOL-v2-real'; please use consistent terminology throughout.
  5. [Figure 2] The dimension annotations in Figure 2 are cluttered and some tensor shapes are ambiguous; please clarify which paths are HW x C and which are C x HW, especially near the transposed attention computation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are external benchmark results, and the only questionable claim (computational cost) is an empirical accounting issue, not a circular derivation.

full rationale

ModalFormer is an empirical system paper. The central claim in Section 4.2 is that the model "surpasses previous methods on all five datasets while reducing computational costs." The PSNR/SSIM and NIQE numbers are benchmark measurements against independent baselines (GLARE, Retinexformer, SNR-Net, etc.) and are not derived from any fitted parameter or prior claim of the authors. The CM-MSA construction (Eqs. 1-3), the FFN (Eq. 4), and the losses (Eqs. 5-6) define an architecture and a training objective; the multimodal loss L_MM supervises the auxiliary subnetworks with 4M-21 features extracted from the normal-light ground-truth references, which is external supervision, not a fitted input renamed as a prediction. The ablation study compares the proposed CM-MSA against plain MSA, additive, concatenation, Q-replacement, and value-scaling variants; these are independent ablated architectures rather than restatements of the final result. The only apparent issue is the "reducing computational costs" clause: the 19.81M parameter total excludes the frozen 4M-21 extractor, and serial inference (728 ms) is slower than the quoted GLARE time (650 ms), with the faster 443 ms figure depending on parallel extraction. This is an under-supported empirical/cost-accounting claim, not a circular derivation, and it does not make any benchmark result equivalent to its input by construction. There is no load-bearing self-citation: 4M-21 [16] is an external model by different authors, and no uniqueness theorem or prior-work-by-the-same-authors premise is invoked to force the architecture. Accordingly, no step in the derivation chain reduces to its own inputs, and the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on external pretrained models (4M-21, CLIP, ImageBind, DINOv2, SAM) providing useful auxiliary features for low-light images, on the definition of ground-truth modalities for the reconstruction loss, and on several hand-chosen or learned fusion parameters (theta_j, tau_j, alpha, beta, gamma, mu). No new physical entities are introduced.

free parameters (4)
  • cross-modal attention balancing weights theta_j = learned
    Per-modality weights in Eq. 3 are learned during training to balance attention maps; they are fitted to the benchmark data.
  • attention temperatures tau_j = learned in (0,1)
    Learnable temperature parameters in Eq. 2 and Eq. 3 for each modality's attention softmax.
  • loss weights alpha, beta, gamma = 0.2, 0.01, 0.1
    Hand-chosen coefficients in Eq. 5 balancing MSE, SSIM, perceptual, and multimodal losses.
  • FFN expansion factor mu = 4
    Hyperparameter for the feed-forward network width, chosen by hand.
assumptions (4)
  • domain assumption 4M-21 extracts eight meaningful 22-channel auxiliary modality feature maps from a low-light image
    The whole approach assumes these features (CLIP, ImageBind, DINOv2, SAM, etc.) are informative for enhancement; no analysis is provided that they contain the required structure.
  • domain assumption Ground-truth auxiliary modalities can be obtained by applying the same extractor to normal-light images, making L_MM well-defined
    The multimodal reconstruction loss compares predicted modalities to ground truth, but the paper never explicitly states that the ground truth is derived from the normal-light reference via 4M-21.
  • domain assumption Pretrained CLIP, ImageBind, DINOv2, SAM features generalize to low-light images
    These models were trained on standard images; their features on dark, noisy inputs are assumed to be useful.
  • domain assumption The product of per-modality attention maps followed by softmax is a valid fusion operation
    This is the core of CM-MSA; the only evidence is the ablation on one dataset, not a theoretical justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ModalFormer: Multimodal Transformer for Low-Light Image Enhancement." pith.science (2026). https://pith.science/paper/XVH3D7OK

@misc{pith2026250720388,
  author       = {Pith},
  title        = {Pith review of: ModalFormer: Multimodal Transformer for Low-Light Image Enhancement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XVH3D7OK}},
  note         = {Machine review of arXiv:2507.20388}
}
read the original abstract

Low-light image enhancement (LLIE) is a fundamental yet challenging task due to the presence of noise, loss of detail, and poor contrast in images captured under insufficient lighting conditions. Recent methods often rely solely on pixel-level transformations of RGB images, neglecting the rich contextual information available from multiple visual modalities. In this paper, we present ModalFormer, the first large-scale multimodal framework for LLIE that fully exploits nine auxiliary modalities to achieve state-of-the-art performance. Our model comprises two main components: a Cross-modal Transformer (CM-T) designed to restore corrupted images while seamlessly integrating multimodal information, and multiple auxiliary subnetworks dedicated to multimodal feature reconstruction. Central to the CM-T is our novel Cross-modal Multi-headed Self-Attention mechanism (CM-MSA), which effectively fuses RGB data with modality-specific features--including deep feature embeddings, segmentation information, geometric cues, and color information--to generate information-rich hybrid attention maps. Extensive experiments on multiple benchmark datasets demonstrate ModalFormer's state-of-the-art performance in LLIE. Pre-trained models and results are made available at https://github.com/albrateanu/ModalFormer.

Figures

Figures reproduced from arXiv: 2507.20388 by the authors.

Figure 1
Figure 1. (a) Our Modality Extractor employs 4M-21 [15, 16] to extract eight auxiliary modalities, with the ninth obtained via the NTSC Luminance Conversion Equation. (b) The overall framework of ModalFormer, showing how CM-T and {U1, . . . , U9} integrate. 3.1 Auxiliary Modalities To enhance the low-light image restoration process, our multimodal framework incorporates nine auxiliary modalities for cross-attention fusion: fe… view at source ↗
Figure 2
Figure 2. The proposed Multimodal Transformer Block (MMTB), which represents the building block of CM- [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Qualitative results on images from LOL-v1 [ [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative results on no-reference datasets [ [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Quantitative comparison with previous multi [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

87 extracted references · 79 canonical work pages

  1. [1]

    Llnet: A deep autoencoder approach to natural low-light image enhancement

    Kin Gwn Lore, Adedotun Akintayo, and Soumik Sarkar. Llnet: A deep autoencoder approach to natural low-light image enhancement. Pattern Recognition, 61:650–662, 2017. 1, 2

    2017

  2. [2]

    Dudhane, S

    A. Dudhane, S. W. Zamir, S. Khan, F. Khan, and M.-H. Yang. Burst image restoration and enhancement. IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2022. 1

    2022

  3. [3]

    S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao. Learning enriched features for real image restoration and enhancement. In European Conference on Computer Vision (ECCV), 2020. 1

    2020

  4. [4]

    R. Liu, L. Ma, J. Zhang, X. Fan, and Z. Luo. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. 1

    2021

  5. [5]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Proceedings of NeurIPS, 2017. 2

    2017

  6. [6]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, and Sylvain Gelly. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations (ICLR) ,

  7. [7]

    Colorization transformer

    Manoj Kumar, Dirk Weissenborn, and Nal Kalchbrenner. Colorization transformer. International Conference on Learning Representations (ICLR), 2021. 2

    2021

  8. [8]

    Learning texture transformer network for image super-resolution

    Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture transformer network for image super-resolution. IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2020. 2

    work page 2020

Show all 87 references
  1. [9]

    Pre-trained image processing transformer

    Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. InIEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2021. 2, 7

    2021

  2. [10]

    S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang. Restormer: Efficient transformer for high-resolution image restoration. In IEEE/CVF Conf. on Computer Vision and Pattern Recog. (CVPR) , 2022. 2, 4

    2022

  3. [11]

    Kim, S.M

    H. Kim, S.M. Choi, C.S. Kim, and Y .J. Koh. Representative color transform for image enhancement.IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 2

    2021

  4. [12]

    Retinexformer: One- stage retinex-based transformer for low-light image enhancement

    Yuanhao Cai, Hao Bian, Jing Lin, Haoqian Wang, Radu Timofte, and Yulun Zhang. Retinexformer: One- stage retinex-based transformer for low-light image enhancement. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 2, 5, 6, 7

    2023

  5. [13]

    W. Su, X. Zhu, Y . Cao, B. Li, L. Lu, F. Wei, and J. Dai. VL-BERT:pre-training of generic visual-linguistic representations. in ArXiv, 2019. 2

    2019

  6. [14]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. Interna- tional Conference on Machine Learning (ICML), 2021. 2, 3

    2021

  7. [15]

    Mizrahi, R

    D. Mizrahi, R. Bachmann, O. F. Kar, T. Yeo, M. Gao, A. Dehghan, and A. Zamir. 4M: Massively multimodal masked modeling. In Advances in Neural Information Processing Systems, 2023. 2, 3

    2023

  8. [16]

    Bachmann, O

    R. Bachmann, O. F. Kar, D. Mizrahi, A. Garjani, M. Gao, D. Griffiths, J. Hu, A. Dehghan, and A. Zamir. 4M-21: An any-to-any vision model for tens of tasks and modalities. In in arXiv, 2024. 2, 3, 7

    2024

  9. [17]

    W. Wang, X. Wu, X. Yuan, and Z. Gao. An experiment-based review of low-light image enhancement methods. IEEE Access, 8:87884–87917, 2020. 2

    2020

  10. [18]

    L. Xiao, C. Li, Z. Wu, and T. Wang. An enhancement method for x-ray image via fuzzy noise removal and homomorphic filtering. Neurocomputing, 195, 2016. 2

    2016

  11. [19]

    S. E. Kim, J. J. Jeon, and I. K. Eom. Image contrast enhancement using entropy scaling in wavelet domain. Signal Processing, 127(1), 2016. 2

    2016

  12. [20]

    Kansal, S

    S. Kansal, S. Purwar, and R. K. Tripathi. Image contrast enhancement using unsharp masking and histogram equalization. Multimedia Tools Applications, 77(20), 2018. 2

    2018

  13. [21]

    Park, S.Yu, B.Moon, S.Ko, and J

    S. Park, S.Yu, B.Moon, S.Ko, and J. Paik. Low-light image enhancement using variational optimization-based retinex model. IEEE Transactions on Consumer Electronics, 63(2), 2017. 2 9

    2017

  14. [22]

    Z. Gu, F. Li, F. Fang, and G. Zhang. A novel retinex-based fractional order variational model for images with severely low light. IEEE Transactions on Image Processing, 29, 2019. 2

    2019

  15. [23]

    B. Cai, X. Xu, K. Guo, K. Jia, B. Hu, and D. Tao. A jointbintrinsic-extrinsic prior model for retinex. IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2017. 2

    2017

  16. [24]

    Deep retinex decomposition for low-light enhance- ment

    Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep retinex decomposition for low-light enhance- ment. In Proceedings of the British Machine Vision Conference (BMVC), 2018. 2, 6, 7, 8

    2018

  17. [25]

    Moran, P

    S. Moran, P. Marza, S. McDonagh, S. Parisot, and G. Slabaugh. DeepLPF: Deep local parametric filters for image enhancement. In IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2020. 2

    2020

  18. [26]

    R. Wang, Q. Zhang, C.-W. Fu, X. Shen, W.-S. Zheng, and J. Jia. Underexposed photo enhancement using deep illumination estimation. In IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2019. 2

    2019

  19. [27]

    Kindling the darkness: A practical low-light image enhancer

    Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo. Kindling the darkness: A practical low-light image enhancer. In ACM International Conference on Multimedia, 2019. 2

    2019

  20. [28]

    Zhang, Y

    Y . Zhang, Y . Tian, Y . Kong, B. Zhong, and Y . Fu. Residual dense network for image restoration. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020. 2

    2020

  21. [29]

    X. Yi, H. Xu, Hao Zhang, Linfeng Tang, and Jiayi Ma. Diff-retinex: Rethinking low-light image enhancement with a generative diffusion model. In IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2023. 2

    2023

  22. [30]

    Enlightengan: Deep light enhancement without paired supervision

    Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang. Enlightengan: Deep light enhancement without paired supervision. IEEE Transactions on Image Processing, 30:2340–2349, 2021. 2, 7

    2021

  23. [31]

    Tokens-to-token vit: Training vision transformers from scratch on imagenet

    Li Yuan, Yunpeng Chen, TaoWang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, , and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. IEEE/CVF Inter- national Conference on Computer Vision (ICCV), 2021. 2

    2021

  24. [32]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 2

    2021

  25. [33]

    Zheng, J

    S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y . Wang, Y . Fu, J. Feng, T. Xiang, and P. HS Torr. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2021. 2

    2021

  26. [34]

    Hierarchical vision transformer using shifted windows

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. Hierarchical vision transformer using shifted windows. European Conference on Computer Vision (ECCV), 2020. 2

    2020

  27. [35]

    Z. Liu, Y . Linand, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 2

    2021

  28. [36]

    X. Xu, R. Wang, C.-W. Fu, and J. Jia. SNR-aware low-light image enhancement. InIEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2022. 2, 6, 7, 8

    2022

  29. [37]

    Zhang, Y

    Z. Zhang, Y . Jiang, J. Jiang, X. Wang, P. Luo, and J. Gu. STAR: A structure-aware lightweight transformer for real-time image enhancement. IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 2

    2021

  30. [38]

    Z. Cui, K. Li, L. Gu, S. Su, P. Gao, Z. Jiang, Y . Qiao, and T. Harada. You only need 90k parameters to adapt light: a light weight transformer for image enhancement and exposure correction. In BMVC, 2022. 2

    2022

  31. [39]

    Z. Wang, X. Cun, J. Bao, W. Zhou, J. Liu, and H. Li. Uformer: A general u-shaped transformer for image restoration. In IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2022. 2

    2022

  32. [40]

    U-Net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention, 2015. 2, 5

    2015

  33. [41]

    W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. Mohammed, S. Singhal, S. Som, and F. Wei. Image as a foreign language: BEiT pretraining for all vision and vision-language tasks. in ArXiv, 2022. 2

    2022

  34. [42]

    Zhang, K

    Y . Zhang, K. Gong, K. Zhang, H. Li, Y . J. Qiao, W. Ouyang, and X. Yue. Meta-transformer: A unified framework for multimodal learning. in ArXiv, 2023. 2

    2023

  35. [43]

    K. He, X. Chen, S. Xie, Y . Li, P. Dollar, and R. Girshick. Masked autoencoders are scalable vision learners. IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2022. 2

    2022

  36. [44]

    Gupta, J

    A. Gupta, J. Wu, J. Deng, and F.F. Li. Siamese masked autoencoders. Proceedings of NeurIPS, 2023. 2 10

    2023

  37. [45]

    Ristea, F.A

    N.C. Ristea, F.A. Croitoru, R.T. Ionescu, M. Popescu, F.S.Khan, and M. Shah. Self-distilled masked auto- encoders are efficient video anomaly detectors. IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR),

  38. [46]

    Georgescu, E

    M.I. Georgescu, E. Fonseca, R.T Ionescu, M. Lucic, C. Schmid, and A. Arnab. Audiovisual masked autoen- coders. IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 2

    2023

  39. [47]

    Zhang, Y

    X. Zhang, Y . Wu, E. Angelini, A. Li, J. Guo, and J.M. Rasmussen. MAPSeg: Unified unsupervised domain adap- tation for heterogeneous medical image segmentation based on 3d masked autoencoding and pseudo-labeling. IEEE/CVF Conf. on Comp. Vision and Pattern Recog. (CVPR), 2024. 2

    2024

  40. [48]

    Y . Wei, A. Gupta, and P. Morgado. Towards latent masked image modeling for self-supervised visual represen- tation learning. European Conference on Computer Vision (ECCV), 2024. 2

    2024

  41. [49]

    H. Qiu, J. Huang, P. Gao, L. Lu, X. Zhang, and S. Lu. Masked autodecoder is effective multi-task vision generalist. IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2024. 2

    2024

  42. [50]

    Z. Tong, Y . Song, J. Wang, and L. Wang. VideoMAE: Masked autoencoders are data-efficient learners for self- supervised video pre-training. Proceedings of NeurIPS, 2022. 2

    2022

  43. [51]

    L. Wang, B. Huang, Z. Zhao, Z. Tong, Y . He, and Y . Wang. VideoMAE V2: Scaling video masked autoencoders with dual masking. IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2023. 2

    2023

  44. [52]

    W. G. C. Bandara, N. Patel, A. Gholami, M. Nikkhah, M. Agrawal, and V . M. Patel. AdaMAE: Adaptive masking for efficient spatiotemporal learning with masked autoencoders. IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2023. 2

    2023

  45. [53]

    H. Yang, D. Huang, B. Wen, J. Wu, H. Yao, Y . Jiang, X. Zhu, and Z. Yuan. Self-supervised video representation learning with motion-aware masked autoencoders. BMVC, 2024. 2

    2024

  46. [54]

    M. Song, M. Yang, W. Wu, D. He, F. Li, and J. Wang. Masked appearance-motion modeling for self-supervised video transformer pretraining. IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2023. 2

    2023

  47. [55]

    L. Yu, Y . Cheng, K. Sohn, J. Lezama, H. Zhang, and H. Chang. MAGVIT: Masked generative video transformer. IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2023. 2

    2023

  48. [56]

    Huang, Z

    W. Huang, Z. Peng, L. Dong, J. Jiao F. Wei, and Q. Ye. Generic to-specific distillation of masked autoencoders. IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2023. 2

    2023

  49. [57]

    H. Chen, J. Gu, Y . Liu, S. A. Magid, C. Dong, Q. Wang, H. Pfister, and L. Zhu. Masked image training for generalizable deep image denoising. IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2023. 2

    2023

  50. [58]

    D. Li, H. Ling, A. Kar, D. Acuna, S. W. Kim, K. Kreis, A. Torralba, and S. Fidler. Dreamteacher: Pretraining image backbones with deep generative models.IEEE/CVF International Conference on Computer Vision (ICCV),

  51. [59]

    Activating more pixels in image super-resolution transformer

    Xiangyu Chen, Xintao Wang, Jiantao Zhou, and Chao Dong. Activating more pixels in image super-resolution transformer. IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2023. 2

    2023

  52. [60]

    DegAE: A new pretraining paradigm for low-level vision

    Yihao Liu, Jingwen He, Jinjin Gu, Xiangtao Kong, Yu Qiao, and Chao Dong. DegAE: A new pretraining paradigm for low-level vision. IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2023. 2

    2023

  53. [61]

    ImageBind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind: One embedding space to bind them all. In IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2023. 3

    2023

  54. [62]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernan- dez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research (T...

    2023

  55. [63]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023. 3

    2023

  56. [64]

    Principal components analysis (pca).Computers & Geosciences, 19(3):303–342, 1993

    Andrzej Ma ´ckiewicz and Waldemar Ratajczak. Principal components analysis (pca).Computers & Geosciences, 19(3):303–342, 1993. 3

    1993

  57. [65]

    S. Yang, D. Zhou, J. Cao, and Y . Guo. Rethinking low-light enhancement via transformer-gan. IEEE Signal Processing Letters, 29:1082–1086, 2022. 5

    2022

  58. [66]

    Sparse gradient regularized deep retinex network for robust low-light image enhancement

    Wenhan Yang, Wenjing Wang, Haofeng Huang, Shiqi Wang, and Jiaying Liu. Sparse gradient regularized deep retinex network for robust low-light image enhancement. IEEE Transactions on Image Processing , 30:2072– 2086, 2021. 6, 7, 8 11

    2021

  59. [67]

    R. Wang, X. Xu, C.-W. Fu, J. Lu, B. Yu, and J. Jia. Seeing dynamic scene in the dark: A high-quality video dataset with mechatronic alignment. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 6, 7

    2021

  60. [68]

    Lime: Low-light image enhancement via illumination map estimation

    Xiaojie Guo, Yu Li, and Haibin Ling. Lime: Low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing, 26(2):982–993, 2016. 6, 7, 8

    2016

  61. [69]

    Naturalness preserved enhancement algorithm for non- uniform illumination images

    Shuhang Wang, Jin Zheng, Hai-Miao Hu, and Bo Li. Naturalness preserved enhancement algorithm for non- uniform illumination images. IEEE Transactions on Image Processing, 22(9):3538–3548, 2013. 6, 7, 8

    2013

  62. [70]

    Perceptual quality assessment for multi-exposure image fusion

    Kede Ma, Kai Zeng, and Zhou Wang. Perceptual quality assessment for multi-exposure image fusion. IEEE Transactions on Image Processing, 24(11):3345–3356, 2015. 6, 7, 8

    2015

  63. [71]

    Contrast enhancement based on layered difference representation of 2d histograms

    Chulwoo Lee, Chul Lee, and Chang-Su Kim. Contrast enhancement based on layered difference representation of 2d histograms. IEEE Transactions on Image Processing, 22(12):5372–5384, 2013. 6, 7, 8

    2013

  64. [72]

    SGDR: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. International Confer- ence on Learning Representations (ICLR), 2017. 6

    2017

  65. [73]

    Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004. 6

    2004

  66. [74]

    completely blind

    Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters, 20(3):209–212, 2012. 6, 7, 8

    2012

  67. [75]

    Y . Wang, R. Wan, W. Yang, H. Li, Lap-Pui Chau, and A. Kot. Low-light image enhancement with normalizing flow. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2604–2612, 2022. 6, 7, 8

    2022

  68. [76]

    H. Zhou, W. Dong, X. Liu, S. Liu, X. Min, G. Zhai, and J. Chen. Glare: Low light image enhancement via generative latent feature based codebook retrieval. In European Conference on Computer Vision (ECCV), 2024. 6, 7, 8

    2024

  69. [77]

    T. Wang, K. Zhang, T. Shen, W. Luo, B. Stenger, and T. Lu. Ultra-high-definition low-light image enhancement: A benchmark and transformer-based method. In AAAI Conference on Artificial Intelligence, 2023. 7

    2023

  70. [78]

    Learning semantic-aware knowledge guidance for low-light image enhancement

    Wu Yuhui, Pan Chen, Wang Guoqing, Yang Yang, Wei Jiwei, Li Chongyi, and Heng Tao Shen. Learning semantic-aware knowledge guidance for low-light image enhancement. In IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2023. 7, 8

    2023

  71. [79]

    Uretinex-net: Retinex- based deep unfolding network for low-light image enhancement

    Wenhui Wu, Jian Weng, Pingping Zhang, Xu Wang, Wenhan Yang, and Jianmin Jiang. Uretinex-net: Retinex- based deep unfolding network for low-light image enhancement. In IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), 2022. 7, 8

    2022

  72. [80]

    You do not need additional priors or regularizers in retinex-based low-light image enhancement

    Huiyuan Fu, Wenkai Zheng, Xiangyu Meng, Xin Wang, Chuanming Wang, and Huadong Ma. You do not need additional priors or regularizers in retinex-based low-light image enhancement. In IEEE/CVF onf. on Computer Vision and Pattern Recog. (CVPR), pages 18125–18134, 2023. 7, 8

    2023

  73. [81]

    Enlighten- your-voice: When multimodal meets zero-shot low-light image enhancement, 2024

    Xiaofeng Zhang, Zishan Xu, Hao Tang, Chaochen Gu, Wei Chen, Shanying Zhu, and Xinping Guan. Enlighten- your-voice: When multimodal meets zero-shot low-light image enhancement, 2024. 8

    2024

  74. [82]

    Implicit neural representation for coop- erative low-light image enhancement

    Shuzhou Yang, Moxuan Ding, Yanmin Wu, Zihan Li, and Jian Zhang. Implicit neural representation for coop- erative low-light image enhancement. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12918–12927, October 2023. 8

    2023

  75. [83]

    Llmra: Multi-modal large language model based restoration assistant, 2024

    Xiaoyu Jin, Yuan Shi, Bin Xia, and Wenming Yang. Llmra: Multi-modal large language model based restoration assistant, 2024. 8

    2024

  76. [84]

    Controlling vision-language models for universal image restoration

    Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sj ¨olund, and Thomas B Sch¨on. Controlling vision-language models for universal image restoration. arXiv preprint arXiv:2310.01018, 2023. 8

    2023 arXiv

  77. [85]

    Geometric-aware low-light image and video enhancement via depth guidance, 2023

    Yingqi Lin, Xiaogang Xu, Yan Han, Jiafei Wu, and Zhe Liu. Geometric-aware low-light image and video enhancement via depth guidance, 2023. 8

    2023

  78. [86]

    Fourllie: Boosting low-light image enhancement by fourier frequency information

    Chenxi Wang, Hongjun Wu, and Zhi Jin. Fourllie: Boosting low-light image enhancement by fourier frequency information. In Proceedings of the 31st ACM International Conference on Multimedia, MM ’23, 2023. 8

    2023

  79. [87]

    Dmfourllie: Dual-stage and multi-branch fourier network for low-light image enhancement

    Tongshun Zhang, Pingping Liu, Ming Zhao, and Haotian Lv. Dmfourllie: Dual-stage and multi-branch fourier network for low-light image enhancement. In Proceedings of the 32nd ACM International Conference on Multi- media, MM ’24, 2024. 8 12

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.