Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

RMMSS: Towards Advanced Robust Multi-Modal Semantic Segmentation with Hybrid Prototype Distillation and Feature Selection

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RMMSS reports a two-stage distillation scheme that lifts missing-modality segmentation by up to 3.89 mIoU while keeping full-modality accuracy within 0.23 of the teacher.

desk verdict A competent two-stage distillation framework with credible robustness gains, but the novelty framing overreaches and the random prototype-pairing choice is never isolated, leaving the core mechanism partly unsupported. read the letter →

arxiv 2505.12861 v2 pith:KZUYRZEM submitted 2025-05-19 cs.CV

Jiaqi Tan , Xu Zheng , Yang Liu This is my paper · ORCID
classification cs.CV
keywords multi-modalsemanticsegmentationmissingmodalityrobustnessknowledgedistillationprototypefeatureselectiondropoutself-distillationcross-modal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a multi-modal segmentation model can be made resilient to missing or corrupted sensor inputs without sacrificing accuracy when all sensors work, by training in two stages. The first stage distills knowledge from a full-modality teacher into a student that is trained with random modality dropout, but the distillation acts on compact class prototypes that are randomly paired across modalities, so each modality learns from the others rather than only from itself. The second stage freezes both the original teacher and the robust student, and a small learned feature-selection module chooses the more informative representation from the two at each location, producing a final student that keeps the robustness while regaining nearly all full-modality accuracy. On the DELIVER, MCubeS, and MUSES benchmarks the reported gains over the prior state of the art are +2.80, +3.89, and +0.89 mIoU in missing-modality tests, with a full-modality drop of only 0.07 to 0.23 mIoU relative to the full-modality teacher.

What carries the argument

The load-bearing object is the class prototype: a vector per semantic class obtained by averaging that class's pixel features (Eq. 4). The Hybrid Prototype Distillation Module (HPDM) compresses student and teacher features into these prototypes and applies a KL-divergence loss between randomly permuted student modality prototypes and teacher modality prototypes (Eq. 5), which is what carries cross-modal knowledge transfer. The Feature Selection Module (FSM) computes a per-location importance score with a depthwise 3x3 convolution plus sigmoid, then selects, from the robust teacher and the full-modality teacher, the features (and logits) with the larger masked value (Eqs. 6–8); this is what lets the final student keep stage-one robustness while recovering full-modality accuracy. Both modules are training-time only, so the final student has the same architecture and parameter count as the baseline.

What would settle it

Retrain stage one with Eq. 5 but replace the random permutation with the identity mapping, so student RGB prototypes are supervised only by teacher RGB prototypes, depth by depth, and so on. If the reported EMM and RMM gains of the full RMMSS framework are unchanged, then the random cross-modal pairing is not the load-bearing mechanism and the paper's explanation for HPDM's benefit fails.

Watch

Extended reading notes

Core claim

The central claim is that robustness to missing modalities and full-modality accuracy are not a trade-off that has to be accepted: they can both be achieved by separating the two goals across stages. Stage one builds a robustness teacher by applying hybrid prototype distillation (HPDM) during self-distillation with modality masking; HPDM turns each modality's features into per-class prototypes and computes KL divergence between randomly paired student and teacher prototypes, which the authors argue is the first use of cross-modal distillation in multi-modal semantic segmentation. Stage two keeps that robust model and adds a Feature Selection Module (FSM) that derives per-location importance scores from the robust teacher and the original full-modality teacher and selects the maximum-valued representation at both feature and logit levels, so the final student learns when to trust robustness versus full detail. Empirically the framework reports missing-modality mIoU gains of +2.80 on DELIVER, +3.89 on MCubeS, and +0.89 on MUSES over the previous best method, while full-modality mIoU falls by only 0.07–0.23 relative to the full-modality teacher; the same wrapper applied to two different backbones yields the same qualitative pattern.

Load-bearing premise

The random pairing in the hybrid prototype distillation assumes that any teacher modality's class prototype is a useful training target for any student modality's prototype, even when the two modalities encode very different scene information.

Editorial extensions

If this is right

  • Any modality-separated multi-modal segmentation backbone can be wrapped in RMMSS and obtain missing-modality robustness with no inference-time overhead, since the added modules are used only during training.
  • The first stage alone surpasses the previous state of the art in robustness, with +2.73 mIoU over AnySeg on the DELIVER arbitrary-modality metric, indicating that cross-modal prototype distillation is the main driver of robustness.
  • Full-modality performance after stage two stays within 0.07–0.23 mIoU of the full-modality teacher, showing that the two-teacher feature selection largely resolves the accuracy-robustness trade-off that plagues single-stage self-distillation.
  • The gains reproduce on three datasets and two backbones (M-SegFormer/AnySeg and CMNeXt), covering RGB-depth-LiDAR-event, material, and adverse-weather sensor settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The random pairing in HPDM can be seen as a cheap stochastic form of cross-modal data augmentation; a natural extension the paper does not test is replacing the uniform random permutation with a curriculum that pairs modalities by measured complementarity, which could sharpen the gain.
  • Because HPDM operates on class prototypes, the same two-stage recipe should transfer to other dense-prediction problems with missing inputs, such as depth estimation or panoptic segmentation, where class-level prototypes can be defined similarly.
  • The FSM's formulation makes no assumption about what the two teachers are; it could be reused to distill an ensemble of full-modality teachers into a single student, or to blend a high-accuracy teacher with a fast teacher, which are testable extensions outside the missing-modality setting.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes RMMSS, a two-stage framework for robust multi-modal semantic segmentation. Stage 1 trains a robust student via self-distillation with modality dropout plus a Hybrid Prototype Distillation Module (HPDM) that computes class prototypes from modality-specific features and applies a random permutation when pairing student and teacher prototypes for cross-modal distillation. Stage 2 freezes a full-modality teacher and the stage-1 robust teacher, then trains a final student with a Feature Selection Module (FSM) that fuses teacher features and logits. Experiments on DELIVER, MCubeS, and MUSES with M-SegFormer and CMNeXt backbones report missing-modality mIoU gains of +2.80%, +3.89%, and +0.89% over AnySeg, while full-modality mIoU drops by only 0.07-0.23% relative to the full-modality teacher.

Significance. The framework is architecture-agnostic, is evaluated on three public benchmarks and two backbone designs, and the ablations in Table 4 separate the contributions of HPDM and FSM. The paper also reports training GPU memory and time, which is useful for practitioners. If the random-pairing mechanism is validated and the module equations are clarified, the two-stage distillation recipe would be a practical contribution to robustness in multi-modal segmentation without sacrificing full-modality accuracy. However, the main novelty currently rests on an unvalidated design choice, and several central equations are underspecified, so the empirical claims are not yet fully reproducible.

major comments (4)
  1. [Method, Eq. (5), Table 4] The central novelty of HPDM is the random permutation pi(m) that pairs student modality pi(m) with teacher modality m. Table 4 only ablates HPDM as a whole, so it does not establish that the random cross-modal pairing, rather than the prototype distillation objective itself, is responsible for the +4.00% EMM gain. Please add ablations with identity pairing (pi(m)=m), a fixed non-identity permutation, and the proposed random permutation, each over several seeds. If identity pairing performs as well as random pairing, the 'hybrid' cross-modal mechanism claimed in contribution (II) is not supported by the evidence.
  2. [Method, Eq. (5)] Equation (5) applies KL divergence to prototype representations p and g of shape [C x d], but KL divergence is defined only for probability distributions. The paper does not specify how these prototype matrices are normalized (e.g., softmax over the class axis or the feature axis), nor how the C x d output is reduced to a scalar. This makes the HPDM loss non-reproducible. Please state the exact tensor shapes, the normalization, and confirm that the loss is differentiable with respect to student features.
  3. [Method, Eqs. (6)-(8), Figure 3] The FSM equations are ambiguous. If ft1 and ft2 each have shape [H,W,C], the concatenated ft should have 2C channels, but Eq. (6) applies DepthwiseConv3x3 with 2 input channels and Eq. (7) reduces from 2 channels to 1. In Eq. (8), phi is described as selecting elements based on the position of the maximum value in m, but after Eq. (7) m is single-channel, so the operation is not a selection between ft1 and ft2. Please clarify the actual channel dimensions and the exact selection operation; as written, the module cannot be implemented unambiguously.
  4. [Experiments, Ablation Study on Hyperparameters, Fig. 6] The hyperparameters lambda and alpha are selected by maximizing the EMM metric, which is also the headline robustness metric reported in Tables 1-3. Unless this tuning is performed on a held-out validation split that is separate from the test set (which is not stated), the reported gains are inflated by selection on the test metric. Please specify the split used for hyperparameter tuning and report the sensitivity of the main results to lambda and alpha over the tested ranges.
minor comments (5)
  1. [Abstract and Introduction] There are typos: 'amd' in the abstract and '0significant' in the introduction; these should be corrected.
  2. [Figure 6] The caption of Figure 6 labels the feature-selection-ratio plot as (a) and the hyperparameter ablation as (b), but the text in 'Analysis of the Effectiveness of FSM' and 'Ablation Study on Hyperparameters' refers to them in the opposite order.
  3. [Appendix references] The text refers to Appendices A, B.1, B.2, C, and D; please ensure all of these are present in the final version, since the main text alone does not define the KL normalization or the FSM tensor operations.
  4. [Table 2] MAGIC++ appears in Table 2 but is not introduced in the results text; add a sentence identifying it and its relation to MAGIC.
  5. [Tables 1-4] All results appear to come from a single run; reporting mean and standard deviation over at least three seeds, or stating that the margins in Tables 1 and 2 exceed run-to-run variability, would strengthen the empirical claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: RMMSS's contributions are evaluated on external benchmarks and no derivation step reduces to its inputs.

full rationale

The paper's derivation chain is empirical rather than definitional. The total loss in Eq. 9 combines a cross-entropy term, a KL distillation term, and the HPDM prototype-distillation term; none of these terms is defined in terms of the EMM, RMM, or Anymodal metrics that the paper later reports. The prototype computation in Eq. 4 is a class-wise feature mean using ground-truth labels, and the random modality pairing in Eq. 5 is a training-time augmentation; neither equation equals the evaluation metric by construction. The central claims about robustness and full-modality preservation are supported by comparisons on three public datasets (DELIVER, MCubeS, MUSES) against multiple external methods, including CMNeXt, MAGIC, MAGIC++, and M-SegFormer. The same-group prior work AnySeg is used as a baseline and as a source of the evaluation protocol, but this is a benchmark choice rather than a load-bearing derivation step: the paper also validates the framework on the CMNeXt backbone and reports module-level ablations (Table 4) that do not depend on AnySeg's fitted values. The hyperparameter search over lambda and alpha using EMM is a model-selection concern, not a circularity, because the reported gains also appear in the HPDM-on/off ablation and in the RMM and full-modality metrics. The absence of an ablation comparing random pairing with identity pairing is a limitation in isolating the cause of the gain, but it does not show that any claimed result is assumed by definition. No uniqueness theorem is imported from the authors' prior work, and no fitted parameter is renamed as a prediction. The paper is therefore self-contained against external empirical evidence, and no significant circularity is present.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper's central claim depends on two hand-tuned loss weights (lambda and alpha), both selected on the EMM robustness metric. It also rests on domain assumptions about teacher-student distillation and on an ad hoc assumption that random cross-modal prototype pairing transfers useful knowledge. No new physical or conceptual entities are introduced.

free parameters (2)
  • lambda (Equation 3) = 50
    Balancing weight for the KL distillation loss L_KL; selected by ablation on the EMM metric in Stage 1.
  • alpha (Equation 5) = 100
    Balancing weight for the hybrid prototype distillation loss L_hp; selected by ablation on the EMM metric.
assumptions (5)
  • domain assumption A full-modality pretrained teacher provides reliable supervision for a student trained with modality dropout.
    Underpins Stage 1 self-distillation; if the teacher is itself biased to full-modality inputs, the student may inherit that bias.
  • ad hoc to paper Random pairing of student and teacher modality prototypes transfers complementary knowledge without injecting noise (Eq. 5).
    Central novelty; no formal justification or sensitivity analysis is provided.
  • domain assumption Ground-truth labels are available to compute class prototypes during training (Eq. 4).
    Prototype extraction uses interpolated labels; this is satisfied in the supervised benchmarks used.
  • domain assumption Normalized teacher features are comparable across modalities, so a convolution-based selection can meaningfully choose between them (Eq. 6).
    FSM assumes unified scale is sufficient for feature selection; no analysis of modality-specific distributions is given.
  • standard math KL divergence between student and teacher softmax outputs is a valid distillation objective (Eq. 2).
    Standard knowledge-distillation loss, widely used and accepted.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RMMSS: Towards Advanced Robust Multi-Modal Semantic Segmentation with Hybrid Prototype Distillation and Feature Selection." pith.science (2026). https://pith.science/paper/KZUYRZEM

@misc{pith2026250512861,
  author       = {Pith},
  title        = {Pith review of: RMMSS: Towards Advanced Robust Multi-Modal Semantic Segmentation with Hybrid Prototype Distillation and Feature Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KZUYRZEM}},
  note         = {Machine review of arXiv:2505.12861}
}
read the original abstract

Multi-modal semantic segmentation (MMSS) faces significant challenges in real-world applications due to incomplete, degraded, or missing sensor data. While current MMSS methods typically use self-distillation with modality dropout to improve robustness, they largely overlook inter-modal correlations and thus suffer significant performance degradation when no modalities are missing. To this end, we present RMMSS, a two-stage framework designed to progressively enhance model robustness under missing-modality conditions, while maintaining strong performance in full-modality scenarios. It comprises two key components: the Hybrid Prototype Distillation Module (HPDM) and the Feature Selection Module (FSM). In the first stage, we pre-train the teacher model with full-modality data and then introduce HPDM to do cross-modal knowledge distillation for obtaining a highly robust model. In the second stage, we freeze both the pre-trained full-modality teacher model and the robust model and propose a trainable FSM that extracts optimal representations from both the feature and logits layers of the models via feature score calculation. This process learns a final student model that maintains strong robustness while achieving high performance under full-modality conditions. Our experiments on three datasets demonstrate that our method improves missing-modality performance by 2.80%, 3.89%, and 0.89%, respectively, compared to the state-of-the-art, while causing almost no drop in full-modality performance (only -0.1% mIoU). Meanwhile, different backbones (AnySeg and CMNeXt) are utilized to validate the generalizability of our framework.

Figures

Figures reproduced from arXiv: 2505.12861 by the authors.

Figure 1
Figure 1. Overview of the RMMSS framework. cross-entropy LCE loss is formulated as follows: LCE = − 1 N X N n=1 X C c=1 yn,clog(P(xn,c)), (1) and the LKL loss is formulated as: LKL = 1 N X N n=1 X C c=1 P(l n,c s )logP(l n,c s ) P(l n,c t ) , (2) where N spans the minibatch dimension, C is the number of classes, P is the softmax function. For Cross-Entropy, x and y are the logits corresponding to class c of the n-th minibatch… view at source ↗
Figure 2
Figure 2. The structure of HPDM. The features f from the teacher and student models are transformed into compact prototype features p via Eq. 5, after which a random match￾ing strategy is applied to p for distillation. refer to Appendix A for details on Data Processing. Hybrid Prototype Distillation Module Prototypical features are more compact than pixel fea￾tures and mine semantic associations in data more effi￾ciently (Wan… view at source ↗
Figure 3
Figure 3. The network design of the FSM Module. The input [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Segmentation results of the model trained with our framework under full-modality and Event missing conditions. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: This chart illustrates the visualization of model [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. J-RAS: Mutual Adaptation for Medical Image Segmentation via Contrastive Retrieval-Augmented Joint Optimization

    cs.CV 2025-10 conditional novelty 5.0 of 10

    Jointly training the retrieval model and the segmentation model improves cardiac MRI segmentation over training the segmentation model alone.

Reference graph

Works this paper leans on

74 extracted references · 58 canonical work pages · cited by 1 Pith paper

  1. [1]

    Y. Lyu, X. Zheng, J. Zhou, and L. Wang, ``Unibind: Llm-augmented unified and balanced representation space to bind them all,'' in CVPR , pp. 26742--26752, IEEE , 2024

    work page 2024

  2. [2]

    Eigen and R

    D. Eigen and R. Fergus, ``Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture,'' in Proceedings of the IEEE international conference on computer vision , pp. 2650--2658, 2015

    work page 2015

  3. [3]

    Y. Liu, Y. Guo, and M. S. Lew, ``On the exploration of convolutional fusion networks for visual recognition,'' in International conference on multimedia modeling , pp. 277--289, Springer, 2016

    work page 2016

  4. [4]

    Zheng, Z

    X. Zheng, Z. Weng, Y. Lyu, L. Jiang, H. Xue, B. Ren, D. Paudel, N. Sebe, L. Van Gool, and X. Hu, ``Retrieval augmented generation and understanding in vision: A survey and new outlook,'' arXiv preprint arXiv:2503.18016 , 2025

    arXiv 2025

  5. [5]

    J. Long, E. Shelhamer, and T. Darrell, ``Fully convolutional networks for semantic segmentation,'' in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 3431--3440, 2015

    work page 2015

  6. [6]

    B. Ren, Y. Liu, Y. Song, W. Bi, R. Cucchiara, N. Sebe, and W. Wang, ``Masked jigsaw puzzle: A versatile position embedding for vision transformers,'' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 20382--20391, 2023

    work page 2023

  7. [7]

    B. Ren, Y. Li, J. Liang, R. Ranjan, M. Liu, R. Cucchiara, L. V. Gool, M.-H. Yang, and N. Sebe, ``Sharing key semantics in transformer makes efficient image restoration,'' Advances in Neural Information Processing Systems , vol. 37, pp. 7427--7463, 2024

    work page 2024

  8. [8]

    Q. Ma, Y. Li, B. Ren, N. Sebe, E. Konukoglu, T. Gevers, L. Van Gool, and D. P. Paudel, ``Shapesplat: A large-scale dataset of gaussian splats and their self-supervised pretraining,'' in International Conference on 3D Vision 2025 , 2024

    work page 2025

Show all 74 references
  1. [9]

    Y. Lyu, X. Zheng, D. Kim, and L. Wang, ``Omnibind: Teach to build unequal-scale modality interaction for omni-bind of all,'' CoRR , vol. abs/2405.16108, 2024

    2024 arXiv

  2. [10]

    Y. Lyu, X. Zheng, and L. Wang, ``Image anything: Towards reasoning-coherent and training-free multi-modal image generation,'' CoRR , vol. abs/2401.17664, 2024

    2024 arXiv

  3. [11]

    C. Wang, C. Wang, W. Li, and H. Wang, ``A brief survey on rgb-d semantic segmentation using deep learning,'' Displays , vol. 70, p. 102080, 2021

    2021

  4. [12]

    Zanuttigh, G

    P. Zanuttigh, G. Marin, C. Dal Mutto, F. Dominio, L. Minto, G. M. Cortelazzo, et al. , ``Time-of-flight and structured light depth cameras,'' Technology and Applications , vol. 978, no. 3, 2016

    2016

  5. [13]

    H. Zhou, L. Qi, Z. Wan, H. Huang, and X. Yang, ``Rgb-d co-attention network for semantic segmentation,'' in Proceedings of the Asian conference on computer vision , 2020

    2020

  6. [14]

    B. Ren, G. Mei, D. P. Paudel, W. Wang, Y. Li, M. Liu, R. Cucchiara, L. Van Gool, and N. Sebe, ``Bringing masked autoencoders explicit contrastive properties for point cloud self-supervised learning,'' in Proceedings of the Asian Conference on Computer Vision , pp. 2034--2052, 2024

    2024

  7. [15]

    Camuffo, D

    E. Camuffo, D. Mari, and S. Milani, ``Recent advancements in learning algorithms for point clouds: An updated overview,'' Sensors , vol. 22, no. 4, p. 1357, 2022

    2022

  8. [16]

    Li and J

    Y. Li and J. Ibanez-Guzman, ``Lidar for autonomous driving: The principles, challenges, and trends for automotive lidar and perception systems,'' IEEE Signal Processing Magazine , vol. 37, no. 4, pp. 50--61, 2020

    2020

  9. [17]

    Padmanabhan, C

    P. Padmanabhan, C. Zhang, and E. Charbon, ``Modeling and analysis of a direct time-of-flight sensor architecture for lidar applications,'' Sensors , vol. 19, no. 24, p. 5464, 2019

    2019

  10. [18]

    J. Zhou, X. Zheng, Y. Lyu, and L. Wang, ``Exact: Language-guided conceptual reasoning and uncertainty estimation for event-based action recognition and more,'' in CVPR , pp. 18633--18643, IEEE , 2024

    2024

  11. [19]

    Alonso and A

    I. Alonso and A. C. Murillo, ``Ev-segnet: Semantic segmentation for event-based cameras,'' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pp. 0--0, 2019

    2019

  12. [20]

    Zhang, K

    J. Zhang, K. Yang, and R. Stiefelhagen, ``Issafe: Improving semantic segmentation in accidents by fusing event-based data,'' in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 1132--1139, IEEE, 2021

    2021

  13. [21]

    Zheng and L

    X. Zheng and L. Wang, ``Eventdance: Unsupervised source-free cross-modal adaptation for event-based object recognition,'' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 17448--17458, 2024

    2024

  14. [22]

    J. Zhou, X. Zheng, Y. Lyu, and L. Wang, ``Eventbind: Learning a unified representation to bind them all for event-based open-world understanding,'' in European Conference on Computer Vision , pp. 477--494, Springer, 2024

    2024

  15. [23]

    J. Zhou, X. Zheng, Y. Lyu, and L. Wang, `` E-CLIP: towards label-efficient event-based open-world understanding by CLIP ,'' CoRR , vol. abs/2308.03135, 2023

    2023 arXiv

  16. [24]

    J. Cao, X. Zheng, Y. Lyu, J. Wang, R. Xu, and L. Wang, ``Chasing day and night: Towards robust and efficient all-day object detection guided by an event camera,'' in ICRA , pp. 9026--9032, IEEE , 2024

    2024

  17. [25]

    Zheng and L

    X. Zheng and L. Wang, ``Eventdance++: Language-guided unsupervised source-free cross-modal adaptation for event-based object recognition,'' CoRR , vol. abs/2409.12778, 2024

    2024 arXiv

  18. [26]

    C. Liao, X. Zheng, Y. Lyu, H. Xue, Y. Cao, J. Wang, K. Yang, and X. Hu, ``Memorysam: Memorize modalities and semantics with segment anything model 2 for multi-modal semantic segmentation,'' arXiv preprint arXiv:2503.06700 , 2025

    2025 arXiv

  19. [27]

    M. K. Reza, A. Prater-Bennette, and M. S. Asif, ``Mmsformer: Multimodal transformer for material and semantic segmentation,'' IEEE Open Journal of Signal Processing , 2024

    2024

  20. [28]

    E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, ``Segformer: Simple and efficient design for semantic segmentation with transformers,'' Advances in neural information processing systems , vol. 34, pp. 12077--12090, 2021

    2021

  21. [29]

    Zhang, H

    J. Zhang, H. Liu, K. Yang, X. Hu, R. Liu, and R. Stiefelhagen, ``Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers,'' IEEE Transactions on intelligent transportation systems , 2023

    2023

  22. [30]

    Zhang, R

    J. Zhang, R. Liu, H. Shi, K. Yang, S. Rei , K. Peng, H. Fu, K. Wang, and R. Stiefelhagen, ``Delivering arbitrary-modal semantic segmentation,'' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 1136--1147, 2023

    2023

  23. [31]

    Zhong, X

    D. Zhong, X. Zheng, C. Liao, Y. Lyu, J. Chen, S. Wu, L. Zhang, and X. Hu, ``Omnisam: Omnidirectional segment anything model for uda in panoramic semantic segmentation,'' arXiv preprint arXiv:2503.07098 , 2025

    2025

  24. [32]

    J. Liu, D. Xu, W. Yang, M. Fan, and H. Huang, ``Benchmarking low-light image enhancement and beyond,'' International Journal of Computer Vision , vol. 129, pp. 1153--1184, 2021

    2021

  25. [33]

    Pfeuffer and K

    A. Pfeuffer and K. Dietmayer, ``Robust semantic segmentation in adverse weather conditions by means of sensor data fusion,'' in 2019 22th International Conference on Information Fusion (FUSION) , pp. 1--8, IEEE, 2019

    2019

  26. [34]

    Q. M. Rahman, N. S \"u nderhauf, P. Corke, and F. Dayoub, ``Fsnet: A failure detection framework for semantic segmentation,'' IEEE Robotics and Automation Letters , vol. 7, no. 2, pp. 3030--3037, 2022

    2022

  27. [35]

    Huang, J

    Y. Huang, J. Lin, C. Zhou, H. Yang, and L. Huang, ``Modality competition: What makes joint training of multi-modal network fail in deep learning?(provably),'' in International conference on machine learning , pp. 9226--9259, PMLR, 2022

    2022

  28. [36]

    Kleinman, A

    M. Kleinman, A. Achille, and S. Soatto, ``Critical learning periods for multisensory integration in deep networks,'' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 24296--24305, 2023

    2023

  29. [37]

    X. Peng, Y. Wei, A. Deng, D. Wang, and D. Hu, ``Balanced multimodal learning via on-the-fly gradient modulation,'' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 8238--8247, 2022

    2022

  30. [38]

    J. Huo, Y. Yan, X. Zheng, Y. Lyu, X. Zou, Z. Wei, and X. Hu, ``Mmunlearner: Reformulating multimodal machine unlearning in the era of multimodal large language models,'' CoRR , vol. abs/2502.11051, 2025

    2025 arXiv

  31. [39]

    Dancette, R

    C. Dancette, R. Cadene, D. Teney, and M. Cord, ``Beyond question-based biases: Assessing multimodal shortcut learning in visual question answering,'' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 1574--1583, 2021

    2021

  32. [40]

    I. Gat, I. Schwartz, A. Schwing, and T. Hazan, ``Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies,'' Advances in Neural Information Processing Systems , vol. 33, pp. 3197--3208, 2020

    2020

  33. [41]

    X. Han, S. Wang, C. Su, Q. Huang, and Q. Tian, ``Greedy gradient ensemble for robust visual question answering,'' in Proceedings of the IEEE/CVF international conference on computer vision , pp. 1584--1593, 2021

    2021

  34. [42]

    Huang, C

    Y. Huang, C. Du, Z. Xue, X. Chen, H. Zhao, and L. Huang, ``What makes multi-modal learning better than single (provably),'' Advances in Neural Information Processing Systems , vol. 34, pp. 10944--10956, 2021

    2021

  35. [43]

    Vosoughi, S

    A. Vosoughi, S. Deng, S. Zhang, Y. Tian, C. Xu, and J. Luo, ``Cross modality bias in visual question answering: A causal view with possible worlds vqa,'' IEEE Transactions on Multimedia , 2024

    2024

  36. [44]

    J. Zhao, F. Teng, K. Luo, G. Zhao, Z. Li, X. Zheng, and K. Yang, ``Unveiling the potential of segment anything model 2 for rgb-thermal semantic segmentation with language guidance,'' CoRR , vol. abs/2503.02581, 2025

    2025 arXiv

  37. [45]

    Zheng, H

    X. Zheng, H. Xue, J. Chen, Y. Yan, L. Jiang, Y. Lyu, K. Yang, L. Zhang, and X. Hu, ``Learning robust anymodal segmentor with unimodal and cross-modal distillation,'' arXiv preprint arXiv:2411.17141 , 2024

    2024 arXiv

  38. [46]

    C. Liao, K. Lei, X. Zheng, J. Moon, Z. Wang, Y. Wang, D. P. Paudel, L. Van Gool, and X. Hu, ``Benchmarking multi-modal semantic segmentation under sensor failures: Missing and noisy modality robustness,'' arXiv preprint arXiv:2503.18445 , 2025

    2025 arXiv

  39. [47]

    Br \"o dermann, D

    T. Br \"o dermann, D. Bruggemann, C. Sakaridis, K. Ta, O. Liagouris, J. Corkill, and L. Van Gool, ``Muses: The multi-sensor semantic perception dataset for driving under uncertainty,'' in European Conference on Computer Vision (ECCV) , 2024

    2024

  40. [48]

    Liang, R

    Y. Liang, R. Wakaki, S. Nobuhara, and K. Nishino, ``Multimodal material segmentation,'' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 19800--19808, June 2022

    2022

  41. [49]

    W. Liu, A. Rabinovich, and A. C. Berg, ``Parsenet: Looking wider to see better,'' arXiv preprint arXiv:1506.04579 , 2015

    2015 arXiv

  42. [50]

    H. Noh, S. Hong, and B. Han, ``Learning deconvolution network for semantic segmentation,'' in Proceedings of the IEEE international conference on computer vision , pp. 1520--1528, 2015

    2015

  43. [51]

    L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, ``Semantic image segmentation with deep convolutional nets and fully connected crfs,'' arXiv preprint arXiv:1412.7062 , 2014

    2014 arXiv

  44. [52]

    L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, ``Encoder-decoder with atrous separable convolution for semantic image segmentation,'' in Proceedings of the European conference on computer vision (ECCV) , pp. 801--818, 2018

    2018

  45. [53]

    T.-Y. Lin, P. Doll \'a r, R. Girshick, K. He, B. Hariharan, and S. Belongie, ``Feature pyramid networks for object detection,'' in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 2117--2125, 2017

    2017

  46. [54]

    Strudel, R

    R. Strudel, R. Garcia, I. Laptev, and C. Schmid, ``Segmenter: Transformer for semantic segmentation,'' in Proceedings of the IEEE/CVF international conference on computer vision , pp. 7262--7272, 2021

    2021

  47. [55]

    Zheng, J

    S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr, et al. , ``Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,'' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognitio...

    2021

  48. [56]

    B. Gao, Y. Pan, C. Li, S. Geng, and H. Zhao, ``Are we hungry for 3d lidar data for semantic segmentation? a survey of datasets and methods,'' IEEE Transactions on Intelligent Transportation Systems , vol. 23, no. 7, pp. 6063--6081, 2021

    2021

  49. [57]

    Y. Zhou, L. Liu, H. Zhao, M. L \'o pez-Ben \' tez, L. Yu, and Y. Yue, ``Towards deep radar perception for autonomous driving: Datasets, methods, and challenges,'' Sensors , vol. 22, no. 11, p. 4208, 2022

    2022

  50. [58]

    Zheng, Y

    X. Zheng, Y. Lyu, L. Jiang, J. Zhou, L. Wang, and X. Hu, ``Magic++: Efficient and resilient modality-agnostic semantic segmentation via hierarchical modality selection,'' arXiv preprint arXiv:2412.16876 , 2024

    2024 arXiv

  51. [59]

    Hinton, O

    G. Hinton, O. Vinyals, and J. Dean, ``Distilling the knowledge in a neural network,'' arXiv preprint arXiv:1503.02531 , 2015

    2015 arXiv

  52. [60]

    Aytar, C

    Y. Aytar, C. Vondrick, and A. Torralba, ``Soundnet: Learning sound representations from unlabeled video,'' Advances in neural information processing systems , vol. 29, 2016

    2016

  53. [61]

    Gupta, J

    S. Gupta, J. Hoffman, and J. Malik, ``Cross modal distillation for supervision transfer,'' in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 2827--2836, 2016

    2016

  54. [62]

    Y. Tian, D. Krishnan, and P. Isola, ``Contrastive multiview coding,'' in Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XI 16 , pp. 776--794, Springer, 2020

    2020

  55. [63]

    Y. Chen, Y. Xian, A. Koepke, Y. Shan, and Z. Akata, ``Distilling audio-visual knowledge by compositional contrastive learning,'' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 7016--7025, 2021

    2021

  56. [64]

    S. Ren, Y. Du, J. Lv, G. Han, and S. He, ``Learning from the master: Distilling cross-modal advanced knowledge for lip reading,'' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 13325--13333, 2021

    2021

  57. [65]

    W. Xia, X. Li, A. Deng, H. Xiong, D. Dou, and D. Hu, ``Robust cross-modal knowledge distillation for unconstrained videos,'' arXiv preprint arXiv:2304.07775 , 2023

    2023 arXiv

  58. [66]

    S. Wang, Z. Yan, D. Zhang, H. Wei, Z. Li, and R. Li, ``Prototype knowledge distillation for medical segmentation with missing modality,'' in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 1--5, IEEE, 2023

    2023

  59. [67]

    Y. Wang, W. Zhou, T. Jiang, X. Bai, and Y. Xu, ``Intra-class feature variation distillation for semantic segmentation,'' in Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part VII 16 , pp. 346--362, Springer, 2020

    2020

  60. [68]

    T. Zhou, W. Wang, E. Konukoglu, and L. Van Gool, ``Rethinking semantic segmentation: A prototype view,'' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 2582--2593, 2022

    2022

  61. [69]

    Z. Wen, G. Xu, M. Tan, Q. Wu, and Q. Wu, ``Debiased visual question answering from feature and sample perspectives,'' Advances in Neural Information Processing Systems , vol. 34, pp. 3784--3796, 2021

    2021

  62. [70]

    S. Zhou, W. Liu, C. Hu, S. Zhou, and C. Ma, ``Unidistill: A universal cross-modality knowledge distillation framework for 3d object detection in bird's-eye view,'' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 5116--5125, 2023

    2023

  63. [71]

    Zheng, Y

    X. Zheng, Y. Lyu, J. Zhou, and L. Wang, ``Centering the value of every modality: Towards efficient and resilient modality-agnostic semantic segmentation,'' in European Conference on Computer Vision , pp. 192--212, Springer, 2024

    2024

  64. [72]

    D. Jia, J. Guo, K. Han, H. Wu, C. Zhang, C. Xu, and X. Chen, ``Geminifusion: Efficient pixel-wise multimodal fusion for vision transformer,'' arXiv preprint arXiv:2406.01210 , 2024

    2024 arXiv

  65. [73]

    B. Li, D. Zhang, Z. Zhao, J. Gao, and X. Li, ``Stitchfusion: Weaving any visual modalities to enhance multimodal semantic segmentation,'' arXiv preprint arXiv:2408.01343 , 2024

    2024 arXiv

  66. [74]

    Zheng, Y

    X. Zheng, Y. Lyu, and L. Wang, ``Learning modality-agnostic representation for semantic segmentation from any modalities,'' in European Conference on Computer Vision , pp. 146--165, Springer, 2025

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.