Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By freezing SAM2's image encoder and training only per-modality LoRA experts with a top-k routing layer, the paper claims MLE-SAM reaches state-of-the-art multi-modal semantic segmentation and gains up to 32.15% over prior methods when…

desk verdict Plausible LoRA-MoE recipe for adapting SAM2 to multi-modal segmentation, but every headline comparison is tainted by a backbone mismatch, so the SOTA claim is unproven. read the letter →

arxiv 2412.04220 v1 pith:QCUPCLNT submitted 2024-12-05 cs.CV cs.AI

Chenyang Zhu , Bin Xiao , Lin Shi , Shoukun Xu , Xu Zheng This is my paper · ORCID
classification cs.CVcs.AI
keywords multi-modalsemanticsegmentationSegmentAnythingModelparameter-efficientfine-tuninglow-rankadaptation(LoRA)mixtureofexpertsfeaturefusionmissingmodalityrobustnessautonomousdriving
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the Segment Anything Model's frozen image encoder can be adapted to multi-modal semantic segmentation (RGB, depth, LiDAR, event, and polarization) by training only small low-rank LoRA experts plus a routing layer. The authors propose MLE-SAM, which assigns each input modality its own LoRA updates inside SAM2's attention while keeping the pre-trained weights frozen, then blends the resulting features with a top-k mixture-of-experts weighting and a dual-pathway mask decoder. They test it on DELIVER, MUSES, and MCubeS and report that it outperforms prior multi-modal segmenters, with particularly large gains when modalities are missing or noisy, such as 32.15% improvement on MUSES under missing-modality testing. If true, this would let foundation-model segmentation be reused across sensor suites without full fine-tuning.

What carries the argument

The mechanism is a Mixture of LoRA Experts (MoE-LoRA) layered on the frozen SAM2 image encoder. Each modality gets its own low-rank update to the attention query and value projections (Eqs. 4-5), so a tiny parameter set adapts the backbone to that sensor. The FPN-style neck produces three feature pyramids per modality: the semantic feature map, the fine-grained pyramid, and the intermediate-resolution pyramid, which are averaged across modalities to form $\mathbf{Y}$, then re-weighted by a softmax top-k router that selects the most relevant modality features at each scale to form $\hat{\mathbf{Y}}$. The two streams are combined and fed into a dual-pathway decoder: SAM2's augmented mask decoder refines logits hierarchically, while an auxiliary MLP-based head fuses all scales; both heads are supervised with OhemCrossEntropy (Eq. 14).

What would settle it

Run CMNeXt, CWSAM, and SAM-LoRA with the same Hiera-B+ backbone and the same 1024x1024 training recipe on DELIVER, MUSES, and MCubeS; if their mIoU rises to within about one point of MLE-SAM, the claimed gains come mostly from the backbone, not the MoE-LoRA routing.

Watch

Extended reading notes

Core claim

The central claim is that SAM2, a model trained for RGB image and video segmentation, can be converted into a strong multi-modal semantic segmenter without updating its weights. The paper argues that inserting modality-specific LoRA matrices into the query and value projections of SAM2's Hiera backbone, together with an equal-weight averaged feature stream, a top-k routing that re-weights the most informative modality features, and an auxiliary segmentation head that fuses multi-scale features, jointly deliver state-of-the-art mIoU on three benchmarks. Evidence includes mIoU of 64.08% on DELIVER with RGB-Depth-Event-LiDAR, 75.42% on MUSES with Frame-LiDAR, and 51.02% on MCubeS with all four material-sensing modalities. The robustness experiments, in which models are trained on all modalities but tested with subsets or with injected Gaussian/random noise, are the basis for the reported 14.13% and 32.15% gains over existing methods.

Load-bearing premise

The load-bearing premise is that comparing MLE-SAM (with the larger Hiera-B+ backbone) against baselines with smaller backbones (MiT-B0 and ViT-B) is a fair test; if the baselines were given the same backbone, the reported mIoU advantages, some as small as 0.43%, could shrink or vanish.

Editorial extensions

If this is right

  • Training only LoRA layers and routing (about 20.8M parameters for four modalities) means a foundation model can serve new sensor configurations without full fine-tuning.
  • If the reported numbers hold, missing or degraded sensors become far less catastrophic: the framework claims 14.13% higher mean mIoU on DELIVER and 32.15% higher on MUSES than prior methods in partial-input tests.
  • The fusion recipe is additive: combining the uniform-average features, the top-k weighted features, and the auxiliary head raises DELIVER mIoU from 58.35% (weighted-only) to 64.08% with the same full model.
  • Dense modalities (RGB, depth, frame) carry most of the performance, while sparse modalities (event, LiDAR) mainly help in adverse conditions; the paper's own tables show sparse-only inputs score near 1-5% mIoU.
  • Because SAM2's weights stay frozen, the approach is compatible with the model's continued use for its original zero-shot segmentation role after the LoRA experts are removed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper: the comparisons give MLE-SAM a larger backbone (Hiera-B+) than the baselines (MiT-B0, ViT-B), so the cleanest test of the MoE-LoRA contribution is a same-backbone rerun; the MCubeS margin over CWSAM is only 0.43%, suggesting the advantage may shrink considerably.
  • Going beyond the paper: the routing weights are trained only on the segmentation loss of the full model, so an untested prediction is that the router's top-k choices should align with sensor quality; one could verify by correlating router weights with per-modality noise levels.
  • Going beyond the paper: the near-catastrophic sparse-modality test scores suggest the model learns RGB/depth priors rather than truly sensor-agnostic features; training with random modality dropout would test whether the reported missing-modality robustness comes from the router or from dense-modality dominance.
  • Going beyond the paper: since only a small parameter set is trained, the same frozen-backbone plus MoE-LoRA recipe should transfer to other foundation encoders or to video/3D perception, provided the same multi-scale fusion heads are added.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MLE-SAM, a method for adapting the SAM2 model to multi-modal semantic segmentation by inserting modality-specific LoRA modules into the frozen image encoder, combining their outputs through a top-k Mixture-of-Experts routing mechanism, and adding an auxiliary segmentation head for multi-scale fusion. The method is evaluated on DELIVER, MUSES, and MCubeS across various modality combinations, including missing-modality and noise-robustness settings, and the authors report consistent mIoU improvements over CMNeXt, CWSAM, and a self-built SAM-LoRA baseline.

Significance. If the reported gains survive a controlled comparison, the paper would make a useful contribution: it demonstrates a parameter-efficient way to adapt a large frozen foundation model to heterogeneous sensor modalities, with a concrete routing mechanism and a dual-head decoder. The experiments cover three benchmarks and several adverse-condition settings, and the method description is sufficiently detailed to be reproduced. The main value is therefore conditional on resolving the backbone-capability confound described below and on providing a same-backbone control that isolates the proposed routing and decoder components.

major comments (4)
  1. [Section IV-A, Tables I-III] The headline comparisons are not controlled for backbone. The paper states in Section IV-A that CMNeXt uses MiT-B0, CWSAM and SAM-LoRA use ViT-B, and MLE-SAM uses Hiera-B+. Because Hiera-B+ is a different, likely stronger encoder than ViT-B, the reported gaps (e.g., 3.39 mIoU over SAM-LoRA on RGB-only in Table I, 8.04 on Frame-only in Table II, and 0.43 on MCubeS RGB-A-D-NIR in Table III) may be due to backbone capacity rather than to the proposed MoE-LoRA or auxiliary head. A same-backbone comparison is required, for instance MLE-SAM with a ViT-B encoder or SAM-LoRA with a Hiera-B+ encoder, before the claim that MLE-SAM 'significantly outperforms' prior methods is supported.
  2. [Section IV-C, Table V] The ablation does not isolate the contribution of the MoE router against a plain per-modality LoRA adaptation on the same backbone. Table V varies integrated features, weighted features, and the auxiliary head, but it does not include a baseline consisting of the frozen Hiera-B+ encoder with per-modality LoRA and simple mask averaging, without the modified SAM2 mask decoder described in Eq. (11). The first row of Table V still uses the modified decoder and the integrated-feature averaging of Eq. (7), so it is not a standard SAM2-LoRA control. Without this control, the specific benefit of the routing mechanism in Eqs. (8)-(10) is not separated from the backbone and decoder changes.
  3. [Section IV-D, Table VII] The abstract's 32.15% missing-modality improvement is misleading because it is measured against CMNeXt, which the paper itself states depends on the Frame/RGB modality and drops to 3.50 mIoU when tested on Frame-only after F-L-E training (Table VII). The gain over the more appropriate same-family baseline SAM-LoRA on the same configuration is 4.99 points, not 32.15. The robustness claim should be reframed relative to baselines that also support training without RGB, and the current wording overstates the magnitude of the benefit.
  4. [Tables I-VIII] No repeated runs or error bars are reported anywhere. Some margins are small (0.43 mIoU on MCubeS RGB-A-D-NIR in Table III, 0.77 mean mIoU over SAM-LoRA on R-D-E-L in Table VI), and without at least three seeds with mean and standard deviation, it is impossible to determine whether these differences are statistically meaningful. The paper should add seed-variability reporting or explicitly discuss the magnitude of run-to-run variance in these benchmarks.
minor comments (5)
  1. [References] Several references appear unrelated to the cited claims: ref. [55] is titled 'Impact of the hubble tension on the r-n s contour', ref. [58] is about anomaly detection for sequential decision sequences, ref. [59] is about orbital dynamics identification, and ref. [60] is about autoformalisation in Lean. These seem to be placeholder or mismatched entries and should be corrected.
  2. [Eq. (10)] The notation in Eq. (10) is ambiguous: 'Top-k(w_1^i,...,w_M^i) ⊙ Y_i^m' suggests that the same top-k mask multiplies every modality feature, but the intent is presumably to zero out or reweight only the selected modalities. Please clarify how the top-k selection is applied across the modality index m.
  3. [Appendix A, Table IX] The loss weights w0 and w1 in Eq. (14) are not listed in the training configuration table. Please specify their values or state that they were set to 1.0.
  4. [Introduction, Section II-B] The statement that this is 'the first attempt to adapt SAM for multi-modal semantic segmentation' is not consistent with the paper's own citation of SAM-Event-Adapter (ref. [50]), which adapts SAM to event-RGB semantic segmentation. Please soften the novelty claim to account for prior adaptation work.
  5. [Figures 5-7] The qualitative figures are difficult to read at the printed resolution; consider enlarging the crops and marking the semantic classes clearly, since the text refers to specific objects such as pedestrians and bus stations.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MLE-SAM's claims rest on external benchmarks and held-out evaluations, not on fitted inputs or load-bearing self-citations.

full rationale

This is an empirical architecture paper with no derivation chain that reduces a predicted quantity to a fitted one. The core comparisons (Tables I–III, VI–VIII) are run on public benchmarks (DELIVER, MUSES, MCubeS) against external methods (CMNeXt, CWSAM) and a self-built SAM-LoRA baseline; the reported mIoU values are held-out test numbers, not re-statements of training objectives. The MoE routing weights (Eqs. 8–10) and LoRA matrices (Eqs. 4–5) are learned parameters, but no result is obtained by fitting a quantity and then 'predicting' that same quantity. The self-citations ([11], [12], [35]) support only background claims about modality-agnostic segmentation and are not load-bearing for any equation or table. The stated backbone standardization ('MiT-B0 was employed for CMNeXt, ViT-B served as the backbone for both CWSAM and SAM-LoRA, while MLE-SAM utilized Hiera-B+') creates a potential confounding of method gains with encoder capacity; that is a fairness/validity concern for the SOTA claim, not a circular reduction of outputs to inputs. Since no step reduces by construction to its own input, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim is empirical, so the ledger is short on fitted constants. The main manually tuned values are the LoRA rank, the per-dataset learning rates, the top-k selection count (unreported), and the loss weights (unreported). The method also assumes that SAM2's RGB-trained encoder can ingest depth, LiDAR, event, and polarization data with only low-rank updates, that global spatial averaging is enough for routing, that the datasets are pre-aligned, and that SAM2's mask decoder can be repurposed for semantic classes.

free parameters (4)
  • LoRA rank = 32
    Set in Appendix Table IX; affects the capacity of modality-specific adaptation and is chosen by hand rather than derived.
  • Top-k selection count k
    Eq. (10) uses top-k modality weighting, but the value of k is never reported; it is a free design choice affecting fusion.
  • Loss weights w0 and w1
    Eq. (14) sums two OhemCrossEntropy losses with scalar weights; the weights are not reported.
  • Per-dataset learning rate = 3e-4 (DELIVER), 6e-4 (MUSES), 8e-3 (MCubeS)
    Chosen per dataset in Appendix Table IX; affects optimization and final performance.
assumptions (5)
  • domain assumption SAM2's RGB-trained Hiera encoder can process depth, LiDAR, event, and polarization inputs with only LoRA updates.
    Section III-C feeds every modality through the same patch embedding and attention backbone; if modalities are not image-like, the premise fails.
  • domain assumption Global spatial averaging in Eq. (8) preserves enough information for modality routing.
    Section III-D computes routing weights from per-level spatial means, discarding spatial structure; informative local regions cannot be distinguished.
  • domain assumption The three benchmark datasets provide pre-aligned multi-modal inputs with consistent sensor geometry.
    Section IV-A describes the datasets but does not discuss alignment or preprocessing beyond resizing and augmentation.
  • domain assumption SAM2's mask decoder can be repurposed for multi-class semantic segmentation.
    Section III-E adapts the instance-segmentation decoder to output class logits; this transfer is assumed to work with the modified heads.
  • domain assumption OhemCrossEntropy with no class weighting is appropriate for the class-imbalanced benchmarks.
    Appendix states the loss is used without class-specific weighting; no analysis supports this choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts." pith.science (2026). https://pith.science/paper/QCUPCLNT

@misc{pith2026241204220,
  author       = {Pith},
  title        = {Pith review of: Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QCUPCLNT}},
  note         = {Machine review of arXiv:2412.04220}
}
read the original abstract

The recent Segment Anything Model (SAM) represents a significant breakthrough in scaling segmentation models, delivering strong performance across various downstream applications in the RGB modality. However, directly applying SAM to emerging visual modalities, such as depth and event data results in suboptimal performance in multi-modal segmentation tasks. In this paper, we make the first attempt to adapt SAM for multi-modal semantic segmentation by proposing a Mixture of Low-Rank Adaptation Experts (MoE-LoRA) tailored for different input visual modalities. By training only the MoE-LoRA layers while keeping SAM's weights frozen, SAM's strong generalization and segmentation capabilities can be preserved for downstream tasks. Specifically, to address cross-modal inconsistencies, we propose a novel MoE routing strategy that adaptively generates weighted features across modalities, enhancing multi-modal feature integration. Additionally, we incorporate multi-scale feature extraction and fusion by adapting SAM's segmentation head and introducing an auxiliary segmentation head to combine multi-scale features for improved segmentation performance effectively. Extensive experiments were conducted on three multi-modal benchmarks: DELIVER, MUSES, and MCubeS. The results consistently demonstrate that the proposed method significantly outperforms state-of-the-art approaches across diverse scenarios. Notably, under the particularly challenging condition of missing modalities, our approach exhibits a substantial performance gain, achieving an improvement of 32.15% compared to existing methods.

Figures

Figures reproduced from arXiv: 2412.04220 by the authors.

Figure 1
Figure 1. (a)Overall of MLE-SAM, (b) Performance on DELIVER (R-D-E-L Modalities), (c) Performance on MUSES (F-E-L [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the proposed MLE-SAM framework for multi-modal semantic segmentation. The architecture combines multi-scale features from a frozen image encoder fine-tuned with LoRA layers. Semantic feature maps and feature pyramids across modalities are averaged and refined via a top-k mechanism. Fused features are processed with a dual-pathway strategy. To achieve an integrated feature representation, we propose a… view at source ↗
Figure 3
Figure 3. Hierarchical Refinement Pathway for High-Resolution [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Visualization of extracted feature maps of DELIVER [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: t-SNE visualization of pixel-level features from selected semantic classes under sensor failure scenarios in the DELIVER [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Comparison of semantic segmentation results on the DELIVER dataset using different methods and modalities [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A multi-modal semantic segmentation framework that processes RGB and non-RGB sensors separately, matches labels in two stages, and aligns cross-modal queries with a VAE refiner.

  2. Partial CLIP is Enough: Chimera-Seg for Zero-shot Semantic Segmentation

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A partial, frozen CLIP block mounted on a segmentation backbone, plus selective distillation to CLIP's CLS token, improves zero-shot semantic segmentation by about 1 hIoU point on two datasets.

  3. MAGIC++: Efficient and Resilient Modality-Agnostic Semantic Segmentation via Hierarchical Modality Selection

    cs.CV 2024-12 reject novelty 5.0 of 10

    MAGIC++ trains a semantic segmentation backbone with all available sensors and then uses the plain backbone at test time, reporting strong average results on arbitrary sensor combinations.

  4. Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation

    cs.CV 2024-11 conditional novelty 5.0 of 10

    AnySeg trains a segmentor to handle arbitrary combinations of visual modalities through unimodal and cross-modal distillation, improving mean mIoU by +6.37% on MUSES and +6.15% on DELIVER over prior state-of-the-art.

  5. EGFormer: Towards Efficient and Generalizable Multimodal Semantic Segmentation

    cs.CV 2025-05 conditional novelty 4.0 of 10

    EGFormer dynamically scores and drops the least useful sensor modality at each processing stage, cutting parameters by up to 91 percent and GFLOPs by half while keeping segmentation accuracy competitive.

Reference graph

Works this paper leans on

62 extracted references · 15 canonical work pages · cited by 5 Pith papers

  1. [55]

    Impact of the hubble tension on the r-n s contour,

    J.-Q. Jiang, G. Ye, and Y .-S. Piao, “Impact of the hubble tension on the r-n s contour,” arXiv preprint arXiv:2303.12345 , Mar. 2023. [Online]. Available: http://arxiv.org/abs/2303.12345v2

    arXiv 2023

  2. [58]

    Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts,

    D. Li, Y . Ma, N. Wang, Z. Ye, Z. Cheng, Y . Tang, Y . Zhang, L. Duan, J. Zuo, C. Yang, and M. Tang, “Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts,” arXiv preprint arXiv:2404.15159 , Apr. 2024. [Online]. Available: http://arxiv.org/abs/2404.15159v3

    arXiv 2024

  3. [59]

    Deep Learning Based Dynamics Identification and Linearization of Orbital Problems using Koopman Theory

    G. Nehma, M. Tiwari, and M. Lingam, “Deep learning based dynamics identification and linearization of orbital problems using koopman theory,” arXiv preprint arXiv:2403.08965 , Mar. 2024. [Online]. Available: http://arxiv.org/abs/2403.08965v2

    work page Pith review arXiv 2024

  4. [60]

    Oil-ad: An anomaly detection framework for sequential decision sequences,

    C. Wang, S. Erfani, T. Alpcan, and C. Leckie, “Oil-ad: An anomaly detection framework for sequential decision sequences,” arXiv preprint arXiv:2402.04567 , Feb. 2024. [Online]. Available: http://arxiv.org/abs/2402.04567v1

    arXiv 2024

  5. [50]

    ClassWise-SAM-Adapter: Parameter Efficient Fine-tuning Adapts Segment Anything to SAR Domain for Semantic Segmentation

    [Online]. Available: http://arxiv.org/abs/2401.02326v1

    work page Pith review arXiv

  6. [1]

    Normalized cuts and image segmentation,

    J. Shi and J. Malik, “Normalized cuts and image segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 22, no. 8, pp. 888–905, 2000. [Online]. Available: https: //doi.org/10.1109/34.868688

    doi:10.1109/34.868688

  7. [2]

    Image segmentation using deep learning: A survey,

    S. Minaee, Y . Y . Boykov, F. Porikli, A. J. Plaza, N. Kehtarnavaz, and D. Terzopoulos, “Image segmentation using deep learning: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 7, pp. 1–1, 2021. [Online]. Available: https://doi.org/10.1109/tpami.2021.3059968

    arXiv 2021

  8. [3]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Doll´ar, and R. Girshick, “Segment anything,” in 2023 IEEE/CVF International Conference on Computer Vision (ICCV) . IEEE, Oct. 2023, pp. 4015–4026. [Online]. Available: https://doi.org/10.1109/iccv51070.2023.00371

    arXiv 2023

Show all 62 references
  1. [4]

    How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with segment anything model,

    H. Gu, H. Dong, J. Yang, and M. A. Mazurowski, “How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with segment anything model,” arXiv preprint arXiv:2404.09957 , Apr. 2024. [Online]. Available: http://arxiv.org/...

    2024 arXiv

  2. [5]

    Medical sam adapter: Adapting segment anything model for medical image segmentation,

    J. Wu, W. Ji, Y . Liu, H. Fu, M. Xu, Y . Xu, and Y . Jin, “Medical sam adapter: Adapting segment anything model for medical image segmentation,” arXiv preprint arXiv:2304.12620 , Apr. 2023. [Online]. Available: http://arxiv.org/abs/2304.12620v7

    2023 arXiv

  3. [6]

    From single- to multi-modal remote sensing imagery interpretation: a survey and taxonomy,

    X. Sun, Y . Tian, W. Lu, P. Wang, R. Niu, H. Yu, and K. Fu, “From single- to multi-modal remote sensing imagery interpretation: a survey and taxonomy,” Science China Information Sciences, vol. 66, no. 4, p. 140301, Mar. 2023. [Online]. Available: https://doi.org/10.1007/s11432...

    2023 doi

  4. [7]

    Ringmo-sam: A foundation model for segment anything in multimodal remote-sensing images,

    Z. Yan, J. Li, X. Li, R. Zhou, W. Zhang, Y . Feng, W. Diao, K. Fu, and X. Sun, “Ringmo-sam: A foundation model for segment anything in multimodal remote-sensing images,” IEEE Transactions on Geoscience and Remote Sensing , vol. 61, pp. 1–16, 2023. [Online]. Available: https://...

    2023

  5. [8]

    Sam 2: Segment anything in images and videos,

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V . Alwala, N. Carion, C.-Y . Wu, R. Girshick, P. Doll ´ar, and C. Feichtenhofer, “Sam 2: Segment anything in images and videos,” arXiv preprint arXiv:...

    2024 arXiv

  6. [10]

    Muses: The multi-sensor semantic perception dataset for driving under uncertainty,

    T. Br ¨odermann, D. Bruggemann, C. Sakaridis, K. Ta, O. Liagouris, J. Corkill, and L. V . Gool, “Muses: The multi-sensor semantic perception dataset for driving under uncertainty,” arXiv preprint arXiv:2401.12761, Jan. 2024. [Online]. Available: http://arxiv.org/abs/2401.12761v4

    2024 arXiv

  7. [11]

    Centering the value of every modality: Towards efficient and resilient modality-agnostic semantic segmentation,

    X. Zheng, Y . Lyu, J. Zhou, and L. Wang, “Centering the value of every modality: Towards efficient and resilient modality-agnostic semantic segmentation,” arXiv preprint arXiv:2407.11344 , Jul. 2024. [Online]. Available: http://arxiv.org/abs/2407.11344v2

    2024 arXiv

  8. [12]

    Learning modality-agnostic representation for semantic segmentation from any modalities,

    X. Zheng, Y . Lyu, and L. Wang, “Learning modality-agnostic representation for semantic segmentation from any modalities,” in Computer Vision – ECCV 2024 . Springer Nature Switzerland, Oct. 2024, pp. 146–165. [Online]. Available: https://doi.org/10.1007/ 978-3-031-72754-2 9

    2024

  9. [15]

    Deep multimodal fusion for semantic image segmentation: A survey,

    Y . Zhang, D. Sidib ´e, O. Morel, and F. M ´eriaudeau, “Deep multimodal fusion for semantic image segmentation: A survey,” Image and Vision Computing, vol. 105, p. 104042, Jan. 2021. [Online]. Available: https://doi.org/10.1016/j.imavis.2020.104042

    2021

  10. [16]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 . Springer International Publishing, 2015, pp. 234–241. [Online]. Available: https://doi.org/...

    2015 doi

  11. [17]

    Segnet: A deep convolutional encoder-decoder architecture for image segmentation,

    V . Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 39, no. 12, pp. 2481–2495, Dec. 2017. [Online]. Available: https://doi.org/...

    2017

  12. [18]

    Unetformer: A unet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery,

    L. Wang, R. Li, C. Zhang, S. Fang, C. Duan, X. Meng, and P. M. Atkinson, “Unetformer: A unet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 190, pp. 196–214, Aug. 2022. [Onli...

    2022 doi

  13. [19]

    Fully convolutional networks for semantic segmentation,

    J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, Jun. 2015, pp. 3431–3440. [Online]. Available: https://doi.org/10.1109/cvpr.2015.7298965

    2015

  14. [20]

    Class-wise fully convolutional network for semantic segmentation of remote sensing images,

    T. Tian, Z. Chu, Q. Hu, and L. Ma, “Class-wise fully convolutional network for semantic segmentation of remote sensing images,” Remote Sensing, vol. 13, no. 16, p. 3211, Aug. 2021. [Online]. Available: https://doi.org/10.3390/rs13163211

    2021 doi

  15. [21]

    Fastfcn: Rethinking dilated convolution in the backbone for semantic segmentation,

    H. Wu, J. Zhang, K. Huang, K. Liang, and Y . Yu, “Fastfcn: Rethinking dilated convolution in the backbone for semantic segmentation,” arXiv preprint arXiv:1903.11816 , Mar. 2019. [Online]. Available: http://arxiv.org/abs/1903.11816v1

    1903 arXiv

  16. [22]

    Rethinking dilated convolution for real-time semantic segmentation,

    R. Gao, “Rethinking dilated convolution for real-time semantic segmentation,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) . IEEE, Jun. 2023, pp. 4675–4684. [Online]. Available: https://doi.org/10.1109/cvprw59228. 2023.00493

    2023

  17. [23]

    Feature pyramid encoding network for real-time semantic segmentation,

    M. Liu and H. Yin, “Feature pyramid encoding network for real-time semantic segmentation,” arXiv preprint arXiv:1909.08599 , Sep. 2019. [Online]. Available: http://arxiv.org/abs/1909.08599v1

    1909 arXiv

  18. [24]

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,

    L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 40, no. 4, pp. 834–848,...

    2018

  19. [25]

    Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,

    S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y . Wang, Y . Fu, J. Feng, T. Xiang, P. H. Torr, and L. Zhang, “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)....

    2021

  20. [26]

    Segmenter: Transformer for semantic segmentation,

    R. Strudel, R. Garcia, I. Laptev, and C. Schmid, “Segmenter: Transformer for semantic segmentation,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . IEEE, Oct. 2021, pp. 7262–7272. [Online]. Available: https://doi.org/10.1109/ iccv48922.2021.00717

    2021

  21. [27]

    Panoptic segformer: Delving deeper into panoptic segmentation with transformers,

    Z. Li, W. Wang, E. Xie, Z. Yu, A. Anandkumar, J. M. Alvarez, P. Luo, and T. Lu, “Panoptic segformer: Delving deeper into panoptic segmentation with transformers,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , vol. 34. IEEE, Jun. 2022, pp. 12 0...

    2022

  22. [28]

    Evaluating transformer- based semantic segmentation networks for pathological image segmentation,

    C. Nguyen, Z. Asad, R. Deng, and Y . Huo, “Evaluating transformer- based semantic segmentation networks for pathological image segmentation,” in Medical Imaging 2022: Image Processing , vol. 12032, SPIE. SPIE, Apr. 2022, p. 128. [Online]. Available: https://doi.org/10.1117/12.2611177

    2022 doi

  23. [29]

    Transunet: Transformers make strong encoders for medical image segmentation,

    J. Chen, Y . Lu, Q. Yu, X. Luo, E. Adeli, Y . Wang, L. Lu, A. L. Yuille, and Y . Zhou, “Transunet: Transformers make strong encoders for medical image segmentation,” arXiv preprint arXiv:2102.04306 , Feb. 2021. [Online]. Available: http://arxiv.org/abs/2102.04306v1

    2021 arXiv

  24. [30]

    Transformer scale gate for semantic segmentation,

    H. Shi, M. Hayat, and J. Cai, “Transformer scale gate for semantic segmentation,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, Jun. 2023, pp. 3051–3060. [Online]. Available: https://doi.org/10.1109/cvpr52729.2023.00298

    2023

  25. [31]

    A survey on deep learning technique for video segmentation,

    T. Zhou, F. Porikli, D. J. Crandall, L. Van Gool, and W. Wang, “A survey on deep learning technique for video segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 6, pp. 7099–7122, Jun. 2023. [Online]. Available: https://doi.org/10.1109/...

    2023

  26. [32]

    A survey on vision transformer,

    K. Han, Y . Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y . Tang, A. Xiao, C. Xu, Y . Xu, Z. Yang, Y . Zhang, and D. Tao, “A survey on vision transformer,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 87–110, Jan. 2023. [Online]. Available: h...

    2023

  27. [33]

    A transformer-based decoder for semantic segmentation with multi-level context mining,

    B. Shi, D. Jiang, X. Zhang, H. Li, W. Dai, J. Zou, H. Xiong, and Q. Tian, “A transformer-based decoder for semantic segmentation with multi-level context mining,” in Computer Vision – ECCV 2022 . Springer Nature Switzerland, 2022, pp. 624–639. [Online]. Available: https://doi....

    2022 doi

  28. [34]

    Transformer-based decoder designs for semantic segmentation on remotely sensed images,

    T. Panboonyuen, K. Jitkajornwanich, S. Lawawirojwong, P. Srestasathiern, and P. Vateekul, “Transformer-based decoder designs for semantic segmentation on remotely sensed images,” Remote Sensing, vol. 13, no. 24, p. 5100, Dec. 2021. [Online]. Available: https://doi.org/10.3390/...

    2021 doi

  29. [35]

    Learning robust anymodal segmentor with unimodal and cross-modal distillation,

    X. Zheng, H. Xue, J. Chen, Y . Yan, L. Jiang, Y . Lyu, K. Yang, L. Zhang, and X. Hu, “Learning robust anymodal segmentor with unimodal and cross-modal distillation,” arXiv preprint arXiv:2411.17141 , Nov. 2024. [Online]. Available: http://arxiv.org/abs/2411.17141v1

    2024 arXiv

  30. [36]

    Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture,

    C. Hazirbas, L. Ma, C. Domokos, and D. Cremers, “Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture,” in Computer Vision – ACCV 2016 . Springer International Publishing, 2017, pp. 213–228. [Online]. Available: https://doi.org/10.1007/978-...

    2016 doi

  31. [37]

    Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks,

    X. Ding, Y . Guo, G. Ding, and J. Han, “Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Oct. 2019, pp. 1911–1920. [Online]. Available: https://doi.org/10.1...

    2019

  32. [38]

    Rtfnet: Rgb-thermal fusion network for semantic segmentation of urban scenes,

    Y . Sun, W. Zuo, and M. Liu, “Rtfnet: Rgb-thermal fusion network for semantic segmentation of urban scenes,” IEEE Robotics and Automation Letters, vol. 4, no. 3, pp. 2576–2583, Jul. 2019. [Online]. Available: https://doi.org/10.1109/lra.2019.2904733

    2019

  33. [39]

    Adapnet: Adaptive semantic segmentation in adverse environmental conditions,

    A. Valada, J. Vertens, A. Dhall, and W. Burgard, “Adapnet: Adaptive semantic segmentation in adverse environmental conditions,” in 2017 IEEE International Conference on Robotics and Automation (ICRA) , IEEE. IEEE, May 2017, pp. 4644–4651. [Online]. Available: https://doi.org/1...

    2017

  34. [40]

    Locality-sensitive deconvolution networks with gated fusion for rgb-d indoor semantic segmentation,

    Y . Cheng, R. Cai, Z. Li, X. Zhao, and K. Huang, “Locality-sensitive deconvolution networks with gated fusion for rgb-d indoor semantic segmentation,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, Jul. 2017, pp. 1475–1483. [Online]. Availabl...

    2017 doi

  35. [41]

    Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data,

    F. I. Diakogiannis, F. Waldner, P. Caccetta, and C. Wu, “Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 162, pp. 94–114, Apr. 2020. [Online]. Available: https://doi.org/10.1016/j...

    2020 doi

  36. [42]

    Cimfnet: Cross-layer interaction and multiscale fusion network for semantic segmentation of high-resolution remote sensing images,

    W. Zhou, J. Jin, J. Lei, and L. Yu, “Cimfnet: Cross-layer interaction and multiscale fusion network for semantic segmentation of high-resolution remote sensing images,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 4, pp. 666–676, Jun. 2022. [Online]. Avai...

    2022

  37. [43]

    Adjacent bi-hierarchical network for scene parsing of remote sensing images,

    J. Ma, W. Zhou, J. Lei, and L. Yu, “Adjacent bi-hierarchical network for scene parsing of remote sensing images,” IEEE Geoscience and Remote Sensing Letters , vol. 20, pp. 1–5, 2023. [Online]. Available: https://doi.org/10.1109/lgrs.2023.3241648

    2023

  38. [44]

    Multimodal remote sensing image segmentation with intuition-inspired hypergraph modeling,

    Q. He, X. Sun, W. Diao, Z. Yan, F. Yao, and K. Fu, “Multimodal remote sensing image segmentation with intuition-inspired hypergraph modeling,” IEEE Transactions on Image Processing , vol. 32, pp. 1474–1487, 2023. [Online]. Available: https://doi.org/10.1109/tip.2023. 3245324

    2023 doi

  39. [45]

    A multilevel multimodal fusion transformer for remote sensing semantic segmentation,

    X. Ma, X. Zhang, M.-O. Pun, and M. Liu, “A multilevel multimodal fusion transformer for remote sensing semantic segmentation,” IEEE Transactions on Geoscience and Remote Sensing , vol. 62, pp. 1–15,

  40. [46]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y . Huang, S.-W. Li, I. Misra, M. Rabbat, V . Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. ...

    2023 arXiv

  41. [47]

    Semantic-sam: Segment and recognize anything at any granularity,

    F. Li, H. Zhang, P. Sun, X. Zou, S. Liu, J. Yang, C. Li, L. Zhang, and J. Gao, “Semantic-sam: Segment and recognize anything at any granularity,” arXiv preprint arXiv:2307.04767 , Jul. 2023. [Online]. Available: http://arxiv.org/abs/2307.04767v1

    2023 arXiv

  42. [49]

    From sam to cams: Exploring segment anything model for weakly supervised semantic segmentation,

    H. Kweon and K.-J. Yoon, “From sam to cams: Exploring segment anything model for weakly supervised semantic segmentation,” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun. 2024, pp. 19 499–19 509. [Online]. Available: https://doi.org/10...

    2024

  43. [51]

    Parameter- efficient fine-tuning for large models: A comprehensive survey,

    Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang, “Parameter- efficient fine-tuning for large models: A comprehensive survey,” arXiv preprint arXiv:2403.14608 , Mar. 2024. [Online]. Available: http://arxiv.org/abs/2403.14608v7

    2024 arXiv

  44. [52]

    Sam-event- adapter: Adapting segment anything model for event-rgb semantic segmentation,

    B. Yao, Y . Deng, Y . Liu, H. Chen, Y . Li, and Z. Yang, “Sam-event- adapter: Adapting segment anything model for event-rgb semantic segmentation,” in 2024 IEEE International Conference on Robotics and Automation (ICRA) , IEEE. IEEE, May 2024, pp. 9093–9100. [Online]. Availabl...

    2024

  45. [53]

    Dylora: Parameter efficient tuning of pre-trained models using dynamic search- free low-rank adaptation,

    M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi, “Dylora: Parameter efficient tuning of pre-trained models using dynamic search- free low-rank adaptation,” arXiv preprint arXiv:2210.07558 , Oct. 2022. [Online]. Available: http://arxiv.org/abs/2210.07558v2

    2022 arXiv

  46. [54]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , Jun. 2021. [Online]. Available: http://arxiv.org/abs/2106.09685v2

    2021 arXiv

  47. [56]

    Sparse low-rank adaptation of pre-trained language models,

    N. Ding, X. Lv, Q. Wang, Y . Chen, B. Zhou, Z. Liu, and M. Sun, “Sparse low-rank adaptation of pre-trained language models,” arXiv preprint arXiv:2311.11696 , Nov. 2023. [Online]. Available: http://arxiv.org/abs/2311.11696v1

    2023 arXiv

  48. [57]

    Adamole: Fine-tuning large language models with adaptive mixture of low-rank adaptation experts,

    Z. Liu and J. Luo, “Adamole: Fine-tuning large language models with adaptive mixture of low-rank adaptation experts,” arXiv preprint arXiv:2405.00361, May 2024. [Online]. Available: http://arxiv.org/abs/ 2405.00361v2

    2024 arXiv

  49. [61]

    Transformer-based visual segmentation: A survey,

    X. Li, H. Ding, H. Yuan, W. Zhang, J. Pang, G. Cheng, K. Chen, Z. Liu, and C. C. Loy, “Transformer-based visual segmentation: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 46, no. 12, pp. 10 138–10 163, Dec. 2024. [Online]. Available: https:/...

    2024

  50. [62]

    Gflean: An autoformalisation framework for lean via gf,

    S. Pathak, “Gflean: An autoformalisation framework for lean via gf,” arXiv preprint arXiv:2404.01234 , Apr. 2024. [Online]. Available: http://arxiv.org/abs/2404.01234v1

    2024 arXiv

  51. [63]

    Training region-based object detectors with online hard example mining,

    A. Shrivastava, A. Gupta, and R. Girshick, “Training region-based object detectors with online hard example mining,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun. 2016, pp. 761–769. [Online]. Available: https://doi.org/10.1109/cvpr.2016.89

    2016 doi

  52. [64]

    Hiera: A hierarchi- cal vision transformer without the bells-and-whistles,

    C. Ryali, Y .-T. Hu, D. Bolya, C. Wei, H. Fan, P.-Y . Huang, V . Aggarwal, A. Chowdhury, O. Poursaeed, J. Hoffman et al. , “Hiera: A hierarchi- cal vision transformer without the bells-and-whistles,” in International Conference on Machine Learning . PMLR, 2023, pp. 29 441–29 454

    2023

  53. [66]

    Decoupled weight decay regularization

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization.” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7 JOURNAL OF ...

    2019

  54. [2024]

    Available: https://doi.org/10.1109/tgrs.2024.3373033

    [Online]. Available: https://doi.org/10.1109/tgrs.2024.3373033

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.