{"id":"ce0d234c-96ed-4184-98d9-b60633425f38","arxiv_id":"2507.07015","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"MST-Distill claims state-of-the-art cross-modal distillation with a mixture of teachers and dynamic routing, but its teacher-masking stage is fit to the student, creating a circular distillation loop.","lead":"Cross-modal knowledge distillation via a mixture of specialized teachers, instance-level routing, and a masking module that adapts teachers to the student. The paper reports accuracy gains across five multimodal datasets, but the teacher-adaptation step is trained to match the student, which risks turning the claimed knowledge transfer into self-training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stage-2 MaskNet adaptation aligns each teacher to the frozen student with no knowledge-preserving objective, so the Stage-3 distillation may be self-distillation and the cross-modal transfer claim is unsupported.","rationale":"The reader's weakest_assumption is exactly the point that makes or breaks the paper. Stage 1 co-trains all models with bidirectional KL, so at the start of Stage 2 the teachers and student are already well aligned. Stage 2 then optimizes MaskNets to close the remaining KL to the student; with no countervailing term, the fixed point is teacher = student. The final distillation then has no external teacher information to transfer, so the method's name and contribution—mixture of specialized teachers for cross-modal knowledge—collapse to a self-distillation regularizer. I also checked other candidate concerns: missing error bars and near-zero increments in Table 1 are real reporting weaknesses, but they are secondary to the mechanism gap. The Grad-CAM and t-SNE figures (Figs. 10–11) are illustrative and do not quantify information preservation. I therefore see no reason to change the reader's REJECT; if the proposed KL measurement were to show that substantial teacher-student divergence survives Stage 2, the verdict would need revisiting, but the current text does not provide that evidence.","tokens_in":19424,"tokens_out":6429,"duration_ms":76603,"concrete_test":"On AV-MNIST, freeze the Stage-1 student and teachers, train Stage 2 as specified, and on a held-out validation split compute the mean KL(P_student ∥ Q_teacher) for each of the N MaskNet-adapted teachers before and after Stage 2. If the post-adaptation KL is below 0.01 nats for essentially all teachers, or if the Stage-3 KD loss in Eq. 14 has near-zero gradient norm relative to the task loss, then the adapted teachers carry no measurable complementary information beyond the student's own output, and the reported accuracy gains must be attributed to self-training or regularization rather than to the mixture of specialized teachers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is MaskNet adaptation (Stage 2). Equation 8/10 define the objective as ℓ_j = KL(Q^j_{m_t} ∥ Q^j_{m_δ(j)}), a pure output-distribution match between each specialized teacher and the frozen target student. Despite the text claiming MaskNet 'reconstructs teacher representations' and 'suppresses modality-specific discrepancies,' no reconstruction term, no discrepancy/complementarity decomposition, and no teacher-fidelity term appears in the loss. A MaskNet that minimizes this objective is optimized to make the teacher's logits mimic the student's logits; the complementary cross-modal information that motivated the mixture is not constrained to survive. Stage 3 then distills these student-shaped teachers back into the same student via Eq. 14, so the supervision is a function of the student itself. The empirical gains in Table 1 are therefore equally explained by self-distillation or regularization, and the paper's core claim that complementary cross-modal knowledge is transferred is not supported. The ablation in Table 3 does not resolve this: it compares pipeline variants, not what information survives MaskNet adaptation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MST-Distill, a three-stage cross-modal knowledge distillation framework. In the first stage, all modality-specific models (one per modality plus a multimodal model) are jointly trained with task and bidirectional KL-alignment losses. In the second stage, MaskNet modules are inserted into the frozen teacher models and trained individually to minimize the KL divergence between each teacher's output and the frozen target-modality student's output. In the third stage, a GateNet routes each sample to the top-k adapted teachers, and the student is trained with a weighted combination of task loss, KL-divergence to the selected teachers' softened outputs, and a load-balancing loss. Experiments are reported on four multimodal classification datasets and on NYU-Depth-V2 for semantic segmentation, together with ablations, routing analyses, and hyperparameter studies. The paper claims that MST-Distill significantly outperforms existing cross-modal knowledge distillation methods.","tokens_in":19613,"tokens_out":6445,"duration_ms":70095,"significance":"If correctly established, the idea of a mixture of specialized teachers with instance-level routing would be a useful contribution to cross-modal knowledge distillation, and the paper has several strengths: it clearly motivates the path-selection and knowledge-drift problems with empirical analyses in Appendix A, it releases code, and it evaluates across five datasets spanning vision, audio, text, and RGB-depth. However, the central mechanism described in Stages 2 and 3 is circular as written: the teachers are adapted to match the frozen student, and the same adapted teacher outputs are then used as distillation targets for that student. In addition, the empirical support for the headline claim is weak because only mean accuracies are reported and several improvements over the strongest baselines are tiny or negative. As a result, the paper does not currently establish that complementary cross-modal knowledge is transferred.","major_comments":[{"comment":"The Stage-2 objective is a pure output-distribution match: ℓ_j = KL(Q^j_{m_t} ∥ Q^j_{m_δ(j)}) minimizes the divergence between each specialized teacher and the frozen target student. There is no term that preserves the teacher's original knowledge, no reconstruction loss for the masked features defined in Eq. (6), and no decomposition of modality-specific discrepancies from complementary cross-modal information. Stage 3 then distills these student-shaped teachers back into the same student via Eq. (14), so the supervision is a function of the student's own outputs. The reported gains are therefore equally explained by self-distillation or regularization, and the paper's central claim that complementary cross-modal knowledge is transferred is not supported.","section":"§3.2, Eq. (8); §3.3, Eq. (14)"},{"comment":"The abstract's claim of 'significantly outperforms existing state-of-the-art' is not verifiable from the reported statistics. Only mean accuracies over five runs are given, with no standard deviations, confidence intervals, or significance tests. Several reported differences against the strongest baseline DML are below 0.5% absolute or negative, for example AV-MNIST Audio (MST-Distill 0.4381 vs DML 0.4393), RAVDESS Audio (0.7174 vs 0.7202), VGGSound Visual (0.4595 vs 0.4601), CrisisMMD Image (0.5495 vs MLLD MM 0.5549), and NYU-Depth-V2 RGB OA (0.5396 vs DML 0.5455). The claimed superiority is therefore not established.","section":"Tables 1 and 2"},{"comment":"The ablation study compares pipeline variants but does not include a self-distillation control in which the frozen student itself, or a teacher exactly matching the student, is used as the distillation target. Such a control is necessary to determine whether the improvements in settings (d), (e), and (f) come from cross-modal knowledge transfer or from the regularization effect of matching the student's own outputs. Without this control, the contribution of MaskNet to cross-modal transfer remains unidentified.","section":"Table 3, ablation study"},{"comment":"The collaborative initialization stage jointly trains all teachers and the student with a bidirectional KL loss and no gradient detachment. This means the 'teachers' are already optimized partly to match the target student's output distribution before the MaskNet adaptation begins. This co-adaptation further confounds the attribution of later gains to complementary cross-modal knowledge, and it should be discussed or controlled for.","section":"§3.1, Eq. (4)"}],"minor_comments":[{"comment":"The text repeatedly states that MaskNet 'reconstructs teacher representations,' but no reconstruction loss is defined; Eq. (6) only produces a masked feature. Please either add a reconstruction objective or rephrase the claims throughout the paper.","section":"§3.2, Eq. (6)"},{"comment":"The notation Q^j_{m_t} is used for the student output in Eq. (8), but Eq. (9) defines Q^j_{m_i} only for teacher outputs f^j_{m_i}; the student's distribution should be defined separately to avoid ambiguity.","section":"§3.2, Eq. (9)"},{"comment":"Some labels in Figure 2 read 'MsakNetT' and appear to be typos for 'MaskNet'; please correct them and ensure the stage names in the figure match the section names.","section":"Figure 2"},{"comment":"The text says MST-Distill shows 'superior performance across all evaluation metrics' and 'ranking first in five of six metrics and second in the remaining one,' but the RGB OA row is below DML; please rephrase to 'best or second-best' for consistency.","section":"Table 2"},{"comment":"Reference [4] contains a typo ('Wjournali Xie'); please proofread the reference list and check all entries for accuracy.","section":"References"},{"comment":"In line 15, the specialized teachers are loaded with Stage-1 base parameters and Stage-2 MaskNet parameters, while the student is loaded from Stage 1; please clarify whether the student is re-initialized or fine-tuned at the start of Stage 3.","section":"Appendix C.1, Algorithm 1"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about MST-Distill is up front: the central mechanism is circular as written. Stage 2 trains each MaskNet to minimize KL divergence from the specialized teacher's output to the frozen student's output (Eq. 8). That is a pure imitation objective—no reconstruction loss, no term that preserves complementary cross-modal information. Then Stage 3 distills those student-shaped teachers back into that same student (Eq. 14). The teacher signal is a function of the student itself, so the reported gains are equally explained by self-distillation or regularization. The text claims MaskNet \"reconstructs teacher representations\" and \"suppresses modality-specific discrepancies,\" but the equations do not do that. The ablation in Table 3 compares pipeline variants; it does not measure what information survives Stage 2. This is a load-bearing flaw, not a minor gap.\n\nThe paper does do some things well. The problem framing—distillation path selection and knowledge drift—is credible and well-motivated by the teacher-transferability experiments in the appendix. The three-stage pipeline with instance-level routing over multiple teachers is a genuinely new combination, and the authors deserve credit for the breadth: five datasets, classification and segmentation, and a code release. The appendix is honest about the heavy training cost (about 4-6x a single-teacher baseline).\n\nThe soft spots beyond the circularity are proportional. Table 1 reports means over five runs with no standard deviations, and several of the headline gains are within 0.5% absolute—on VGGSound visual and RAVDESS audio, MST-Distill actually trails DML. The abstract's \"significantly outperforms\" is not supported by the reported numbers. The method may still work as a regularizer, and the routing dynamics in Figure 4 are interesting, but the paper never establishes that complementary cross-modal knowledge is transferred.\n\nWho is this for? A reader working on cross-modal distillation or teacher ensembling might find the routing and multi-teacher setup worth a look, but they should treat the mechanism claim with caution. This is a solid engineering paper in need of a major reinterpretation. It deserves a serious referee—the problem is important, the experimental setup is thorough, and the flaw is addressable in principle—but the current submission does not support its core claim. My recommendation: send it to review, expect heavy revision, and don't take the cross-modal transfer claim at face value.","headline":"The mixture-of-teachers routing is a fresh engineering idea, but the MaskNet adaptation step makes teachers imitate the student, so the paper's cross-modal transfer claim is not supported and the empirical gains lack statistical backing.","tokens_in":20192,"tokens_out":2528,"would_cite":false,"duration_ms":29091,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cross-modal knowledge distillation improves when each training sample is routed to its own best teacher, and a mask module retunes that teacher to the student's behavior.","keywords":["cross-modal knowledge distillation","mixture of teachers","dynamic routing","MaskNet","knowledge drift","multimodal learning","teacher-student training"],"falsifier":"Run a control on a dataset with a strong teacher advantage (for example the visual student on RAVDESS): replace MaskNet with direct fine-tuning of the frozen teacher's logits toward the student under the same KL loss and compute budget, then compare transfer gains; if the two gains are statistically indistinguishable, the adapted teacher is a noisy copy of the student and the reported improvement is self-training rather than knowledge transfer.","tokens_in":19183,"feed_emoji":"🎓","tokens_out":3727,"duration_ms":43886,"temperature":0.7,"pith_summary":"This paper tries to establish that cross-modal knowledge distillation fails for two fixable reasons: the teacher is chosen once and statically, and the teacher's inductive bias drifts from the student's so its guidance is partly unusable. The proposed answer is MST-Distill, a three-stage framework in which a pool of multimodal and cross-modal teachers is first trained together, each teacher is then specialized toward the target student by a learnable MaskNet that re-filters its internal features, and finally a per-sample routing network lets the student dynamically pick the top teacher for each input. The authors argue this matters because real deployments often run a single-modality student (for example a vision-only edge model) that could benefit from knowledge in other modalities without paying the cost of full multimodal inference. They report that on five multimodal datasets, spanning classification and semantic segmentation, this adaptive pipeline consistently beats single-teacher and static-teacher distillation baselines.","feed_headline":"Each sample picks its best teacher in cross-modal distillation","feed_subtitle":"A routing network plus mask-based teacher retuning beats fixed-teacher distillation on five multimodal datasets.","key_machinery":"The load-bearing machinery is the mixture of specialized teachers: a pool of $N$ teachers formed by inserting independent MaskNet modules into selected layers of the multimodal and auxiliary unimodal models, plus a GateNet router that scores all $N$ teachers from the student's logits and selects the top $k$. MaskNet itself is a multi-head self-attention block followed by a linear layer and sigmoid, producing a soft mask that is multiplied onto intermediate teacher features; it is trained alone, with all base-model parameters frozen, to minimize the KL divergence between the specialized teacher's softened output and the student's output. The argument leans on two identities being separable: that the routing network can learn which teacher transfers best for a given sample, and that MaskNet can remove only the teacher's modality-specific discrepancies while preserving its complementary cross-modal knowledge.","core_discovery":"The paper's central claim is that no fixed teacher, whether multimodal or cross-modal, is reliably the best source of supervision for a target-modality student: transferability is asymmetric across modality directions, varies per dataset, and even varies per instance. MST-Distill therefore replaces the static teacher with a mixture of specialized teachers and an instance-level router. In the first stage, all modality-specific models are trained jointly with task loss and bidirectional KL alignment, without gradient detachment, to create aligned starting points. In the second stage, independently parameterized MaskNet modules are inserted at chosen layers of each frozen teacher and trained only against the student's softened outputs, with the stated purpose of suppressing modality-specific discrepancies and reconstructing teacher representations that match the student's behavior. In the third stage, the student's own logits are fed to a small routing network whose top-$k$ teacher selection (by default $k=1$) determines which specialized teacher supervises each sample, with a load-balancing loss keeping the full pool in use. The paper reports that this design outperforms existing response-, feature-, and relation-based distillation methods as well as recent cross-modal methods on AV-MNIST, RAVDESS, VGGSound-50k, CrisisMMD-V2, and NYU-Depth-V2.","pith_inferences":["Editorial extension: the routing mechanism is not tied to modality; the same GateNet-plus-MaskNet recipe could be tested on a pool of same-modality teachers with different architectures, where per-sample selection might absorb model-capacity or initialization differences instead of modality differences.","Editorial extension: routing probabilities over training (reported for RAVDESS) could be compared against an oracle that, for each validation sample, picks the teacher whose distillation gave the largest student improvement; the gap would quantify how much headroom remains in the router.","Editorial extension: the paper leaves implicit that MaskNet's per-sample reconstruction could act as an instance-level curriculum, gradually steering easy samples toward one teacher and hard samples toward another; a testable prediction is that routing entropy should be higher on hard samples early in training and should decrease over time."],"forward_implications":["A student trained under MST-Distill should be able to absorb cross-modal knowledge even when the static choice of a single teacher would fail, because the router can pick a multimodal teacher for some samples and a cross-modal teacher for others.","The stage ordering matters: the paper's ablations show that dynamic distillation applied directly to static teachers yields minimal gains, implying that collaborative initialization and MaskNet specialization are prerequisites for the router to be useful.","The framework transfers beyond classification to dense prediction, since the same three stages with pixel-level distillation improve both RGB and depth students on NYU-Depth-V2.","With all teachers selected at once ($k=4$), routing degrades into uniform averaging and performance drops, so the per-sample selection mechanism itself, not mere teacher diversity, is credited with the gains."],"supporting_citations":[{"why":"Provides the classical logits-based distillation objective (softened-output KL divergence) that MST-Distill adapts for its routing and MaskNet losses.","marker":"[15]"},{"why":"C2KD supplies the main recent cross-modal baseline and the experimental protocol that MST-Distill follows, and motivates modality imbalance and dynamic sample selection.","marker":"[17]"},{"why":"The modality focusing hypothesis motivates both the MGDFR baseline MST-Distill beats and the feature-significance filtering idea that MaskNet generalizes.","marker":"[44]"},{"why":"MKE establishes the earlier generalized framework for cross-modal knowledge exchange between unimodal and multimodal models that MST-Distill extends.","marker":"[45]"},{"why":"Defines unimodal bias in multimodal deep networks, which the paper identifies as the source of knowledge drift between teacher and student attention.","marker":"[50]"},{"why":"Deep mutual learning is the competitive mutual-learning baseline and the conceptual ancestor of the collaborative initialization stage's bidirectional alignment.","marker":"[51]"},{"why":"Supplies the multi-head self-attention block that MaskNet is built on for generating its soft feature mask.","marker":"[38]"}],"fun_headline_variants":["Mixture of teachers, routed per sample, beats fixed distillation","Per-instance teacher selection improves cross-modal distillation","MST-Distill: Router picks the best specialist teacher for each sample","Adaptive teacher routing outperforms static cross-modal distillation","Don't use one teacher: MST-Distill routes each sample to its best"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim stands on the premise that MaskNet, trained only to make the teacher's outputs match the student's, suppresses modality-specific discrepancies while leaving the teacher's complementary cross-modal knowledge intact; the training signal itself cannot distinguish those two things.","fun_headline_variants_meta":{"raw":{"variants":["Mixture of teachers, routed per sample, beats fixed distillation","Per-instance teacher selection improves cross-modal distillation","MST-Distill: Router picks the best specialist teacher for each sample","Adaptive teacher routing outperforms static cross-modal distillation","Don't use one teacher: MST-Distill routes each sample to its best"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1474,"prompt_tokens":1013,"completion_tokens":461,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":373}},"tokens_in":629,"tokens_out":461,"duration_ms":5421,"temperature":1.0,"reasoning_tokens":373,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:49:26.927392+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a control on a dataset with a strong teacher advantage (for example the visual student on RAVDESS): replace MaskNet with direct fine-tuning of the frozen teacher's logits toward the student under the same KL loss and compute budget, then compare transfer gains; if the two gains are statistically indistinguishable, the adapted teacher is a noisy copy of the student and the reported improvement is self-training rather than knowledge transfer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"C2KD supplies the main recent cross-modal baseline and the experimental protocol that MST-Distill follows, and motivates modality imbalance and dynamic sample selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The modality focusing hypothesis motivates both the MGDFR baseline MST-Distill beats and the feature-significance filtering idea that MaskNet generalizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MKE establishes the earlier generalized framework for cross-modal knowledge exchange between unimodal and multimodal models that MST-Distill extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines unimodal bias in multimodal deep networks, which the paper identifies as the source of knowledge drift between teacher and student attention."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Deep mutual learning is the competitive mutual-learning baseline and the conceptual ancestor of the collaborative initialization stage's bidirectional alignment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the multi-head self-attention block that MaskNet is built on for generating its soft feature mask."}],"review_version":1}