{"id":"0bb101af-6a26-4c56-b6c7-2386c2745006","arxiv_id":"2505.12861","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RMMSS improves missing-modality segmentation mIoU by up to 3.89% on public benchmarks while keeping full-modality mIoU within 0.1% of a full-modality teacher.","lead":"What did this paper find or do: It presents a two-stage training framework that makes multi-modal semantic segmentation models more accurate when one or more sensors fail, while keeping their accuracy nearly unchanged when all sensors work. Why might a smart generalist read it: Sensor failures are common in real autonomous driving, so a method that makes segmentation more robust without sacrificing normal performance is practically useful.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HPDM's random cross-modal pairing is untested against identity pairing; the claimed hybrid mechanism may not be the source of the robustness gain.","rationale":"The reader's weakest assumption correctly identifies Eq. 5's random permutation as the core untested mechanism. I agree that this is the single most load-bearing concern because HPDM is the primary novel component claimed to enhance robustness, yet its specific design choice (random cross-modal pairing) is not isolated in any ablation. Table 4 only shows HPDM on/off, conflating the prototype distillation operation with the random pairing strategy. A simple identity-pairing baseline would clarify whether the 'hybrid' aspect contributes anything beyond same-modality distillation. This matters for the central claim: if the gain comes solely from prototype-based self-distillation, the paper's framing as cross-modal hybrid distillation is misleading, and the method reduces to a variant of existing self-distillation. Other concerns, such as the false 'first to introduce cross-modal distillation' claim (AnySeg's title explicitly includes cross-modal distillation), no code/error bars, and hyperparameter tuning on EMM, are noted but are secondary to the mechanistic validity of the central contribution. My concern does not invalidate the reported empirical results; it questions the interpretation. Therefore, the conditional verdict remains appropriate, pending the proposed ablation and code release.","tokens_in":12530,"tokens_out":6515,"duration_ms":64108,"concrete_test":"Run a controlled ablation on DELIVER with the M-SegFormer backbone, stage 1 only, comparing three variants of Eq. 5: (a) random permutation pi as in the paper, (b) identity pairing pi(m)=m (same-modality prototype distillation), and (c) all-pairs cross-modal KL (each student prototype matched to every teacher prototype). Keep all other hyperparameters and training settings identical. If variant (b) achieves EMM within 0.5% of variant (a), the random-pairing hypothesis is not supported and the 'hybrid' claim should be revised; if (a) clearly outperforms (b), the assumption holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that RMMSS improves missing-modality robustness rests on the Hybrid Prototype Distillation Module (HPDM), whose loss (Eq. 5) randomly permutes student modality prototypes before computing KL divergence against teacher prototypes. This implicitly assumes that any teacher modality's prototype is a valid supervision target for any student modality's prototype. No evidence is provided that modality-specific feature spaces are aligned well enough for this random pairing to transfer complementary knowledge rather than inject noise. The only supporting ablation (Table 4) turns HPDM on/off as a whole; it does not isolate the random-pairing choice. If identity pairing (matching each student modality to its corresponding teacher modality) gives the same robustness gain, then the 'hybrid' cross-modal mechanism is not the cause, and the paper's stated contribution (II) is unsupported. Conversely, if random pairing significantly outperforms identity pairing, the assumption is validated. This is load-bearing because HPDM is the primary novelty responsible for the stage-1 robustness improvement (+10.5% EMM in Table 3), and the full-modality preservation in stage 2 builds on that robustness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RMMSS, a two-stage framework for robust multi-modal semantic segmentation. Stage 1 trains a robust student via self-distillation with modality dropout plus a Hybrid Prototype Distillation Module (HPDM) that computes class prototypes from modality-specific features and applies a random permutation when pairing student and teacher prototypes for cross-modal distillation. Stage 2 freezes a full-modality teacher and the stage-1 robust teacher, then trains a final student with a Feature Selection Module (FSM) that fuses teacher features and logits. Experiments on DELIVER, MCubeS, and MUSES with M-SegFormer and CMNeXt backbones report missing-modality mIoU gains of +2.80%, +3.89%, and +0.89% over AnySeg, while full-modality mIoU drops by only 0.07-0.23% relative to the full-modality teacher.","tokens_in":12741,"tokens_out":6328,"duration_ms":65436,"significance":"The framework is architecture-agnostic, is evaluated on three public benchmarks and two backbone designs, and the ablations in Table 4 separate the contributions of HPDM and FSM. The paper also reports training GPU memory and time, which is useful for practitioners. If the random-pairing mechanism is validated and the module equations are clarified, the two-stage distillation recipe would be a practical contribution to robustness in multi-modal segmentation without sacrificing full-modality accuracy. However, the main novelty currently rests on an unvalidated design choice, and several central equations are underspecified, so the empirical claims are not yet fully reproducible.","major_comments":[{"comment":"The central novelty of HPDM is the random permutation pi(m) that pairs student modality pi(m) with teacher modality m. Table 4 only ablates HPDM as a whole, so it does not establish that the random cross-modal pairing, rather than the prototype distillation objective itself, is responsible for the +4.00% EMM gain. Please add ablations with identity pairing (pi(m)=m), a fixed non-identity permutation, and the proposed random permutation, each over several seeds. If identity pairing performs as well as random pairing, the 'hybrid' cross-modal mechanism claimed in contribution (II) is not supported by the evidence.","section":"Method, Eq. (5), Table 4"},{"comment":"Equation (5) applies KL divergence to prototype representations p and g of shape [C x d], but KL divergence is defined only for probability distributions. The paper does not specify how these prototype matrices are normalized (e.g., softmax over the class axis or the feature axis), nor how the C x d output is reduced to a scalar. This makes the HPDM loss non-reproducible. Please state the exact tensor shapes, the normalization, and confirm that the loss is differentiable with respect to student features.","section":"Method, Eq. (5)"},{"comment":"The FSM equations are ambiguous. If ft1 and ft2 each have shape [H,W,C], the concatenated ft should have 2C channels, but Eq. (6) applies DepthwiseConv3x3 with 2 input channels and Eq. (7) reduces from 2 channels to 1. In Eq. (8), phi is described as selecting elements based on the position of the maximum value in m, but after Eq. (7) m is single-channel, so the operation is not a selection between ft1 and ft2. Please clarify the actual channel dimensions and the exact selection operation; as written, the module cannot be implemented unambiguously.","section":"Method, Eqs. (6)-(8), Figure 3"},{"comment":"The hyperparameters lambda and alpha are selected by maximizing the EMM metric, which is also the headline robustness metric reported in Tables 1-3. Unless this tuning is performed on a held-out validation split that is separate from the test set (which is not stated), the reported gains are inflated by selection on the test metric. Please specify the split used for hyperparameter tuning and report the sensitivity of the main results to lambda and alpha over the tested ranges.","section":"Experiments, Ablation Study on Hyperparameters, Fig. 6"}],"minor_comments":[{"comment":"There are typos: 'amd' in the abstract and '0significant' in the introduction; these should be corrected.","section":"Abstract and Introduction"},{"comment":"The caption of Figure 6 labels the feature-selection-ratio plot as (a) and the hyperparameter ablation as (b), but the text in 'Analysis of the Effectiveness of FSM' and 'Ablation Study on Hyperparameters' refers to them in the opposite order.","section":"Figure 6"},{"comment":"The text refers to Appendices A, B.1, B.2, C, and D; please ensure all of these are present in the final version, since the main text alone does not define the KL normalization or the FSM tensor operations.","section":"Appendix references"},{"comment":"MAGIC++ appears in Table 2 but is not introduced in the results text; add a sentence identifying it and its relation to MAGIC.","section":"Table 2"},{"comment":"All results appear to come from a single run; reporting mean and standard deviation over at least three seeds, or stating that the margins in Tables 1 and 2 exceed run-to-run variability, would strengthen the empirical claims.","section":"Tables 1-4"}],"recommendation":"major_revision","confidential_remarks":"The SoTA baseline AnySeg (Zheng et al. 2024c) shares a co-author with this submission, but the manuscript does not disclose this relationship. The editor should ask the authors to state it and to confirm that the AnySeg comparison uses the official implementation and protocol. This is a transparency issue rather than a technical flaw."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a solid empirical paper on robust multi-modal segmentation, with consistent gains across three benchmarks and two backbones. But the \"first to introduce cross-modal distillation into MMSS\" claim is wrong — AnySeg, which they cite and whose author overlaps, already does unimodal and cross-modal distillation. And the stress-test concern is legitimate: the random-pairing choice in HPDM is load-bearing and untested.\n\nWhat's genuinely good: the two-stage design is clean. Stage 1 trains a robust teacher via modality-masked self-distillation plus prototype distillation; stage 2 freezes both teachers and uses a feature-selection module to fuse their outputs. The empirical story holds together: +2.8 mIoU on DELIVER, +3.89 on MCubeS, +0.89 on MUSES over AnySeg, with full-modality drop around 0.1%. The CMNeXt experiments help generalizability, and the ablations show both modules contribute. That's meaningful incremental value for the robustness subfield.\n\nSoft spots, in proportion:\n\n1. Random pairing untested. Eq. 5 pairs student modality π(m) with teacher modality m via random permutation. The paper never compares against identity pairing. If identity pairing gives the same gain, the \"hybrid\" cross-modal mechanism is not the source. Table 4 turns HPDM on/off, which doesn't answer this. This is a one-experiment fix but as written it's a real gap.\n\n2. Hyperparameters tuned on the evaluation metric. λ and α are selected on EMM, then EMM gains are the headline. That's a mild selection-on-eval issue; they should acknowledge it or validate on a hold-out failure split.\n\n3. No error bars or multiple seeds. The statement \"random seeds are fixed... error margins within 0.1%\" is not the same as reporting variance. Given the gains are several points, this is less critical, but still worth noting.\n\n4. FSM equations are ambiguous. DepthwiseConv3×3(2,2) with two channels is odd, and the masking function φ is loosely described. Implementation details are deferred to an appendix.\n\n5. Novelty framing. Position as a new prototype-level distillation objective, not the first cross-modal distillation in MMSS.\n\nCitation pattern looks fine. No code is mentioned, which matters for reproducibility.\n\nWho this is for: researchers working on sensor-failure robustness in multi-modal segmentation. It deserves a serious referee, not a desk reject; the empirical results are concrete and the ablations are informative. The authors should be asked to run the identity-vs-random pairing ablation and to clarify the hyperparameter selection. I'd send it to peer review with those requests.","headline":"A competent two-stage distillation framework with credible robustness gains, but the novelty framing overreaches and the random prototype-pairing choice is never isolated, leaving the core mechanism partly unsupported.","tokens_in":13243,"tokens_out":2157,"would_cite":false,"duration_ms":23262,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RMMSS reports a two-stage distillation scheme that lifts missing-modality segmentation by up to 3.89 mIoU while keeping full-modality accuracy within 0.23 of the teacher.","keywords":["multi-modal semantic segmentation","missing modality robustness","knowledge distillation","prototype distillation","feature selection","modality dropout","self-distillation","cross-modal distillation"],"falsifier":"Retrain stage one with Eq. 5 but replace the random permutation with the identity mapping, so student RGB prototypes are supervised only by teacher RGB prototypes, depth by depth, and so on. If the reported EMM and RMM gains of the full RMMSS framework are unchanged, then the random cross-modal pairing is not the load-bearing mechanism and the paper's explanation for HPDM's benefit fails.","tokens_in":12303,"feed_emoji":"🚗","tokens_out":7394,"duration_ms":70560,"temperature":0.7,"pith_summary":"This paper claims that a multi-modal segmentation model can be made resilient to missing or corrupted sensor inputs without sacrificing accuracy when all sensors work, by training in two stages. The first stage distills knowledge from a full-modality teacher into a student that is trained with random modality dropout, but the distillation acts on compact class prototypes that are randomly paired across modalities, so each modality learns from the others rather than only from itself. The second stage freezes both the original teacher and the robust student, and a small learned feature-selection module chooses the more informative representation from the two at each location, producing a final student that keeps the robustness while regaining nearly all full-modality accuracy. On the DELIVER, MCubeS, and MUSES benchmarks the reported gains over the prior state of the art are +2.80, +3.89, and +0.89 mIoU in missing-modality tests, with a full-modality drop of only 0.07 to 0.23 mIoU relative to the full-modality teacher.","feed_headline":"Two-stage distillation lifts missing-sensor segmentation by ~3.9 mIoU","feed_subtitle":"Full-modality accuracy drops only ~0.1 mIoU while sensor-failure cases gain up to 3.89.","key_machinery":"The load-bearing object is the class prototype: a vector per semantic class obtained by averaging that class's pixel features (Eq. 4). The Hybrid Prototype Distillation Module (HPDM) compresses student and teacher features into these prototypes and applies a KL-divergence loss between randomly permuted student modality prototypes and teacher modality prototypes (Eq. 5), which is what carries cross-modal knowledge transfer. The Feature Selection Module (FSM) computes a per-location importance score with a depthwise 3x3 convolution plus sigmoid, then selects, from the robust teacher and the full-modality teacher, the features (and logits) with the larger masked value (Eqs. 6–8); this is what lets the final student keep stage-one robustness while recovering full-modality accuracy. Both modules are training-time only, so the final student has the same architecture and parameter count as the baseline.","core_discovery":"The central claim is that robustness to missing modalities and full-modality accuracy are not a trade-off that has to be accepted: they can both be achieved by separating the two goals across stages. Stage one builds a robustness teacher by applying hybrid prototype distillation (HPDM) during self-distillation with modality masking; HPDM turns each modality's features into per-class prototypes and computes KL divergence between randomly paired student and teacher prototypes, which the authors argue is the first use of cross-modal distillation in multi-modal semantic segmentation. Stage two keeps that robust model and adds a Feature Selection Module (FSM) that derives per-location importance scores from the robust teacher and the original full-modality teacher and selects the maximum-valued representation at both feature and logit levels, so the final student learns when to trust robustness versus full detail. Empirically the framework reports missing-modality mIoU gains of +2.80 on DELIVER, +3.89 on MCubeS, and +0.89 on MUSES over the previous best method, while full-modality mIoU falls by only 0.07–0.23 relative to the full-modality teacher; the same wrapper applied to two different backbones yields the same qualitative pattern.","pith_inferences":["The random pairing in HPDM can be seen as a cheap stochastic form of cross-modal data augmentation; a natural extension the paper does not test is replacing the uniform random permutation with a curriculum that pairs modalities by measured complementarity, which could sharpen the gain.","Because HPDM operates on class prototypes, the same two-stage recipe should transfer to other dense-prediction problems with missing inputs, such as depth estimation or panoptic segmentation, where class-level prototypes can be defined similarly.","The FSM's formulation makes no assumption about what the two teachers are; it could be reused to distill an ensemble of full-modality teachers into a single student, or to blend a high-accuracy teacher with a fast teacher, which are testable extensions outside the missing-modality setting."],"forward_implications":["Any modality-separated multi-modal segmentation backbone can be wrapped in RMMSS and obtain missing-modality robustness with no inference-time overhead, since the added modules are used only during training.","The first stage alone surpasses the previous state of the art in robustness, with +2.73 mIoU over AnySeg on the DELIVER arbitrary-modality metric, indicating that cross-modal prototype distillation is the main driver of robustness.","Full-modality performance after stage two stays within 0.07–0.23 mIoU of the full-modality teacher, showing that the two-teacher feature selection largely resolves the accuracy-robustness trade-off that plagues single-stage self-distillation.","The gains reproduce on three datasets and two backbones (M-SegFormer/AnySeg and CMNeXt), covering RGB-depth-LiDAR-event, material, and adverse-weather sensor settings."],"supporting_citations":[{"why":"Provides AnySeg, the state-of-the-art self-distillation baseline and the arbitrary-modality evaluation protocol that RMMSS extends and compares against.","marker":"Zheng et al. 2024c"},{"why":"Establishes the teacher-student knowledge distillation paradigm that both training stages build on.","marker":"Hinton, Vinyals, and Dean 2015"},{"why":"Introduces prototype knowledge distillation for segmentation with missing modalities, directly motivating HPDM's prototype-level transfer.","marker":"Wang et al. 2023"},{"why":"Supplies the prototype view of semantic segmentation used to justify compressing features to class prototypes before distillation.","marker":"Zhou et al. 2022"},{"why":"Contributes the DELIVER dataset and the CMNeXt backbone, serving as the primary benchmark and one of the two validation backbones.","marker":"Zhang et al. 2023b"},{"why":"Provides the MCubeS multi-modal material segmentation dataset used in the second benchmark.","marker":"Liang et al. 2022"},{"why":"Provides the MUSES multi-sensor adverse-driving dataset used in the third benchmark.","marker":"Brödermann et al. 2024"},{"why":"Supplies robust cross-modal knowledge distillation techniques cited as inspiration for the hybrid distillation strategy.","marker":"Xia et al. 2023"}],"fun_headline_variants":["Missing-sensor mIoU rises 3.89, while full-modality drops only 0.1","RMMSS gains 3.89 on missing sensors, loses just 0.1 mIoU in full","Two-stage distillation lifts missing-sensor mIoU by 3.89, full drops 0.1","Robust MMSS without trade-off: missing gains 3.89, full loses only 0.1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The random pairing in the hybrid prototype distillation assumes that any teacher modality's class prototype is a useful training target for any student modality's prototype, even when the two modalities encode very different scene information.","fun_headline_variants_meta":{"raw":{"variants":["Missing-sensor mIoU rises 3.89, while full-modality drops only 0.1","RMMSS gains 3.89 on missing sensors, loses just 0.1 mIoU in full","Two-stage distillation lifts missing-sensor mIoU by 3.89, full drops 0.1","Robust MMSS without trade-off: missing gains 3.89, full loses only 0.1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002401,"raw_usage":{"total_tokens":9309,"prompt_tokens":1092,"completion_tokens":8217,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":708,"completion_tokens_details":{"reasoning_tokens":8105}},"tokens_in":708,"tokens_out":8217,"duration_ms":54962,"temperature":1.0,"reasoning_tokens":8105,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:24:48.985386+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain stage one with Eq. 5 but replace the random permutation with the identity mapping, so student RGB prototypes are supervised only by teacher RGB prototypes, depth by depth, and so on. If the reported EMM and RMM gains of the full RMMSS framework are unchanged, then the random cross-modal pairing is not the load-bearing mechanism and the paper's explanation for HPDM's benefit fails.","supporting_citations":[],"review_version":1}