{"id":"00d885ea-a8ea-4a90-be00-8b91d0ce9469","arxiv_id":"2607.27054","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Calibrating teacher–student feature decorrelation with semantic confusion weights, classifier-induced strength maps, and adaptive loss scaling beats uniform redundancy suppression on heterogeneous KD.","lead":"CoCaRS improves cross-architecture knowledge distillation by calibrating which feature correlations to suppress, instead of decorrelating them uniformly. It matters for deploying smaller models when the best teacher has a different architecture than the student.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"SAC/CEE structure-vs-redundancy proxies remain the load-bearing soft spot; ImageNet support is thin without uncertainty.","rationale":"The reader already isolated the right hinge: SAC’s classifier-subspace strength map and CEE’s retrieval confusion weights are the assumptions that must hold for “retain structure, suppress redundancy” to explain the gains over uniform RSD. I did not find a deeper internal inconsistency in the loss algebra (Eqs. 1–2 vs 10–15) or a result that contradicts the tables; ablations and the inverted-SAC control are genuine supporting evidence, and homogeneous gains (Table 7) plus CKA (Fig. 2) are consistent with better alignment. The residual risk is mechanistic attribution plus thin ImageNet margins without uncertainty—exactly why CONDITIONAL (code, multi-seed, fuller hyperparams) is already the right bar. No verdict shift: keep CONDITIONAL with high confidence in what was measured and medium correctness risk on why.","tokens_in":15820,"tokens_out":658,"duration_ms":53926,"concrete_test":"Re-evaluate Table 2 (all four ImageNet pairs) for RSD vs CoCaRS with three independent seeds; report mean±std. Separately, on Swin-T→ResNet18 (CIFAR-100), run CoCaRS with M=1 bank using only the distillation teacher’s logits (no extra teachers) and with SAC map sign-flipped only on the top quartile of M_off. If the ImageNet mean gap is <0.2% or CIs overlap on ≥3/4 pairs, or if M=1 / quartile-flip erases most of the +1.5pt CIFAR gain over RSD, the calibration-as-structure narrative (and large-scale claim) weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim attributes CIFAR/ImageNet gains over RSD to calibrated decorrelation that keeps structural correlations while suppressing redundancy (SCC: CEE Eqs. 5–8 + SAC Eq. 9), plus ACR. That story requires two proxies to be correctly signed: (1) off-diagonal mass in M_sem=|QQᵀ| (QR basis of normalized teacher classifier weights) marks structure that should receive weaker decorrelation, and (2) retrieval-bank confusion weights are reliable sample importance for correlation estimation. Ablations (Tables 3–5) show removing or inverting these pieces hurts, but that only shows the knobs are useful regularizers—not that they separate structure from redundancy. Neural-collapse motivation is ambiguous: dimensions that co-load on the same classifier direction may be redundant copies of one discriminative signal (favoring stronger decorrelation) rather than complementary structure (favoring weaker). CEE further injects a prebuilt multi-teacher response bank and key-encoder retrieval, so gains may partly reflect auxiliary dark-knowledge aggregation rather than pure RSD calibration. ImageNet average lift is only +0.43% with no error bars or multi-seed stats (Table 2), so the large-scale half of the claim is especially sensitive to this mechanism gap.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes CoCaRS for heterogeneous knowledge distillation, refining RSD-style redundancy suppression. It keeps diagonal cross-architecture invariance in the teacher–student Pearson correlation matrix while calibrating off-diagonal decorrelation via Semantic Correlation Calibration (SCC): Confusion Evidence Estimation (CEE) builds sample confusion weights from positive/reciprocal-negative teacher dark knowledge in a retrieval bank (Eqs. 5–8), and Strength Allocation Control (SAC) builds a semantic strength map from a QR-induced discriminative subspace of normalized teacher classifier weights so stronger associations receive weaker decorrelation (Eq. 9). Adaptive Coefficient Regulation (ACR) then sets the SCC coefficient from the relative SCC/CE loss scale via EMA (Eqs. 13–15). On CIFAR-100 and ImageNet-1K across CNN/ViT/MLP pairs, CoCaRS reports higher Top-1 than RSD and other heterogeneous KD baselines (CIFAR avg. 84.70% vs RSD 82.14%; ImageNet avg. 74.46% vs 74.03%), with component and formulation ablations, homogeneous transfer, CKA similarity, and coefficient-scale plots.","tokens_in":16223,"tokens_out":1586,"duration_ms":31005,"significance":"If the gains hold under stronger statistical reporting and clearer mechanism checks, CoCaRS is a useful incremental advance on redundancy-suppression heterogeneous KD: it targets two practical RSD weaknesses (uniform off-diagonal decorrelation and fixed β) with a modular calibration stack and broad multi-architecture evaluation. Strengths include systematic ablations (Tables 3–6), homogeneous-setting results (Table 7), CKA analysis (Fig. 2), compute/memory comparison (Fig. 3), and explicit ACR dynamics (Fig. 4). The work is empirical rather than theoretical; significance rests on reliable large-scale gains and on whether CEE/SAC truly separate structure from redundancy rather than acting as useful but opaque regularizers. Code release is promised but not yet available.","major_comments":[{"comment":"SAC’s signed structure-vs-redundancy assumption is load-bearing but under-justified. Method (SAC, Eq. 9) sets M_κ ∝ exp(−τ_κ M_off) with M_sem = |QQᵀ| from QR on normalized teacher classifier weights, so larger off-diagonal subspace association lowers decorrelation. Neural-collapse motivation is ambiguous: co-loading dimensions may be redundant copies of one discriminative direction (favoring stronger decorrelation) rather than complementary structure. Table 5 shows Inverted ≈ w/o SAC and Random worse than full SAC, which establishes usefulness of the knob and preferred sign under this recipe, not that M_off identifies structural correlations that should be preserved. A direct diagnostic is needed (e.g., controlled synthetic redundancy, or measuring whether protected correlations improve class-separability / NC geometry rather than only final accuracy).","section":"Method, Strength Allocation Control; Table 5"},{"comment":"CEE may confound “calibrated RSD” with auxiliary multi-teacher dark-knowledge aggregation. Eqs. 4–8 build a response bank over pretrained teachers {T_m}, retrieve neighbors with a key encoder, and reweight correlation estimation by w_conf. Gains attributed to correlation calibration could partly come from this external supervision channel rather than better structure/redundancy separation inside P. Table 4’s Random retrieval drop helps, but does not isolate bank/multi-teacher logits from pure sample reweighting of the existing teacher–student pair. Please ablate single-teacher bank vs multi-teacher, bank-free CEE (e.g., using only the current teacher’s non-target logits), and report whether SCC still beats RSD without retrieval infrastructure.","section":"Method, Confusion Evidence Estimation; Table 4"},{"comment":"ImageNet support for the central claim is thin relative to the CIFAR story. Table 2 average lift over RSD is +0.43% (one pair −0. something on Mixer→MobileNetV2 where CoCaRS 72.15 is below PAT 72.22), with no error bars, seeds, or significance tests anywhere in Tables 1–2. The strongest claim couples CIFAR and ImageNet improvements plus reduced coefficient sensitivity; without multi-seed uncertainty, the large-scale half is not yet commensurate with the mechanism narrative. At minimum report mean±std over ≥3 seeds on ImageNet pairs and clarify hyperparameter selection for κ, ρ, α, γ, τ_κ.","section":"Table 2; Experiments"},{"comment":"ACR’s claim to “reduce sensitivity to coefficient settings” is only partially evidenced. Fig. 4 shows relative SCC/CE scale varies across pairs and that λ_t is adapted, but there is no head-to-head sensitivity sweep (fixed λ grid vs ACR) reporting accuracy variance across coefficients/pairs/stages—the quantity the abstract and introduction emphasize. Without that comparison (or a table of performance under mismatched fixed β/λ), the sensitivity-reduction claim remains qualitative. Please add a coefficient-sensitivity experiment on at least two heterogeneous pairs.","section":"Abstract; Adaptive Coefficient Regulation; Fig. 4"}],"minor_comments":[{"comment":"Eq. (1) formatting is hard to parse (missing clear fraction bars/parentheses for the Pearson denominator). Please re-typeset P_ij in standard form.","section":"Preliminaries, Eq. (1)"},{"comment":"Fig. 4 caption/text inconsistency: body says “After ARC” in one place while the method is ACR; fix typo.","section":"Fig. 4"},{"comment":"Several free parameters (κ, ρ, η, G(·), α, γ, τ_κ, retrieval design, stable-rank target) are introduced with limited default values or selection protocol in the main text. A compact hyperparameter table would aid reproducibility pending code release.","section":"Method; Experiments"},{"comment":"Related Work could more sharply separate CoCaRS from SemCKD / SimKD / NCKD beyond motivation, since CEE and classifier-subspace use overlap thematically with those lines.","section":"Related Work"},{"comment":"Table 1 DIST entry for Swin-T→ResMLP-S12 (11.05%) looks like a training failure; flag or explain outliers so averages are not skewed without comment.","section":"Table 1"},{"comment":"Promise that “Code will be released soon” should be paired with a concrete artifact plan (configs, bank construction, seeds) given the retrieval-bank dependency.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"Solid empirical KD engineering with broad pair coverage; not a conceptual breakthrough beyond RSD. Main risk is over-claiming mechanism (structure-preserving calibration) when ablations only show regularizer utility, plus small ImageNet margins without uncertainty. Major revision is appropriate: fixable with diagnostics, isolation ablations, multi-seed ImageNet, and a real coefficient-sensitivity study. Scope fits a solid ML systems/methods venue more than a theory-heavy venue."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a careful systems paper on top of RSD, not a new paradigm. They keep the teacher–student correlation matrix and diagonal invariance, then stop treating every off-diagonal entry the same—CEE reweights samples via retrieved teacher dark knowledge, SAC softens decorrelation where the teacher classifier’s QR subspace says dimensions are associated, and ACR EMA-scales the term against CE so a fixed λ hurts less across pairs and epochs.\n\nWhat is actually new is that combination and the empirical package around it. CIFAR-100 is the strong half: average 84.70% vs RSD 82.14%, wins on every listed heterogeneous pair, and the component/formulation ablations (remove CEE/SAC/ACR, drop negative evidence, invert SAC, random retrieval/subspace, don’t touch the diagonal) all move the right way. Homogeneous ImageNet pairs and the CKA figure are consistent extras. Related work is honest about Barlow/VICReg, neural collapse, SemCKD/SimKD, and RSD; they are not pretending to invent redundancy reduction.\n\nSoft spots, in proportion. The load-bearing story—that M_sem and confusion weights separate “structure” from “redundancy”—is only weakly identified. Ablations show the knobs help; they do not prove the semantic interpretation. Neural-collapse geometry could equally mean those dimensions are redundant copies of one signal, which would argue for stronger, not weaker, decorrelation. CEE also brings a multi-teacher response bank and key encoder, so some lift may be auxiliary dark-knowledge aggregation rather than pure RSD calibration. ImageNet average is +0.43% with no error bars or multi-seed stats; that half of the claim is thin. Free parameters (κ, ρ, η, α, γ, τ_κ, retrieval design) are real, and code is still “soon.” None of that sinks the paper; it caps how hard you should lean on the mechanism narrative.\n\nWho it is for: people already doing heterogeneous KD or compression across CNN/ViT/MLP. Worth a serious referee. I would engage, cite it as the stronger RSD-style baseline once code lands, and not treat the structure-vs-redundancy story as settled without more probes.","headline":"Solid incremental fix on RSD: calibrated off-diagonal decorrelation plus loss-scale balancing, with clear CIFAR gains and thinner ImageNet support.","tokens_in":16817,"tokens_out":559,"would_cite":true,"duration_ms":15636,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Calibrating which feature correlations to suppress lets small models learn better from teachers with different architectures.","keywords":["knowledge distillation","heterogeneous architectures","redundancy suppression","feature correlation","correlation calibration","model compression","adaptive loss weighting"],"falsifier":"If ablating CEE and SAC (reverting to uniform off-diagonal penalties) or replacing the classifier-derived strength map and retrieved confusion bank with random counterparts closes the accuracy gap to RSD on the same heterogeneous CIFAR-100 and ImageNet pairs, the calibration claim fails.","tokens_in":16693,"feed_emoji":"🔗","tokens_out":869,"duration_ms":20167,"temperature":0.7,"pith_summary":"When a compact student network is trained from a larger teacher with a different architecture, their internal features often disagree in ways that blunt ordinary knowledge distillation. Prior redundancy-suppression methods force teacher–student feature correlations toward identity, keeping cross-architecture invariance on the diagonal while zeroing off-diagonal correlations as pure redundancy. This paper argues that uniform zeroing also erases useful structure, and that a fixed loss weight makes the method brittle across model pairs and training stages. CoCaRS instead calibrates decorrelation: confusion evidence from teacher responses reweights which samples matter for estimating correlations, a strength map built from the teacher classifier’s discriminative subspace softens decorrelation where structure should be kept, and an adaptive coefficient keeps the calibrated term balanced against the task loss. On CIFAR-100 and ImageNet-1K across CNN, ViT, and MLP pairs, the approach raises accuracy over uniform redundancy suppression and prior heterogeneous distillation baselines while reducing sensitivity to the coefficient.","feed_headline":"Smarter feature decorrelation lifts cross-architecture distillation","feed_subtitle":"Protect structural correlations, adapt the loss weight, and students beat uniform redundancy suppression.","key_machinery":"Semantic Correlation Calibration (SCC): the off-diagonal part of the teacher–student correlation objective is reweighted by confusion-derived sample strengths (CEE) and a classifier-subspace strength map (SAC), with Adaptive Coefficient Regulation (ACR) setting the SCC weight from the running SCC-to-task loss ratio.","core_discovery":"Uniform off-diagonal decorrelation in teacher–student feature correlations is too blunt for heterogeneous knowledge distillation: some correlations encode structural information that should be protected. CoCaRS improves distillation by calibrating that decorrelation—via sample-level confusion weights from teacher dark knowledge and a semantic strength map from the teacher classifier subspace—while adaptively regulating the term’s contribution from its loss scale relative to cross-entropy, yielding higher student accuracy and lower coefficient sensitivity than fixed uniform redundancy suppression.","pith_inferences":["The same protect-structure-while-decorrelating idea may transfer to other cross-architecture alignment settings beyond classification KD, such as detection or multi-modal towers with mismatched backbones.","If classifier-subspace maps are the right structural prior mainly near neural-collapse regimes, calibration quality may track how collapsed the teacher is, suggesting a simple diagnostic before distillation.","Building a retrieval bank of teacher logits adds a preprocessing step; cheaper online approximations of confusion evidence would test how much of the gain needs the offline bank."],"forward_implications":["Heterogeneous distillation should treat off-diagonal feature correlations as mixed structure-plus-redundancy, not pure noise to zero uniformly.","Teacher classifier geometry and non-target logits can serve as practical guides for how hard to decorrelate each feature pair.","Loss-scale adaptive weighting can replace brittle fixed coefficients when a distillation auxiliary term changes magnitude across architectures and epochs.","Students distilled this way should show higher intermediate-feature similarity to heterogeneous teachers, especially where uniform RSD still leaves a gap."],"fun_headline_variants":["Calibrated decorrelation protects structure in heterogeneous KD","CoCaRS tunes redundancy suppression for cross-architecture distillation","Adaptive correlation calibration improves heterogeneous KD","Protecting structural correlations beats uniform decorrelation","Confusion-weighted decorrelation lifts cross-architecture students"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The method assumes that teacher-response confusion scores and associations in the teacher classifier’s feature subspace correctly mark which correlations are structure to keep rather than redundancy to kill.","fun_headline_variants_meta":{"raw":{"variants":["Calibrated decorrelation protects structure in heterogeneous KD","CoCaRS tunes redundancy suppression for cross-architecture distillation","Adaptive correlation calibration improves heterogeneous KD","Protecting structural correlations beats uniform decorrelation","Confusion-weighted decorrelation lifts cross-architecture students"]},"model":"grok-4.5","effort":"low","cost_usd":0.002978,"raw_usage":{"total_tokens":1084,"prompt_tokens":827,"num_sources_used":0,"completion_tokens":53,"cost_in_usd_ticks":29784000,"prompt_tokens_details":{"text_tokens":827,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":204,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":827,"tokens_out":53,"duration_ms":4826,"temperature":1.0,"reasoning_tokens":204,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T12:33:41.640239+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"If ablating CEE and SAC (reverting to uniform off-diagonal penalties) or replacing the classifier-derived strength map and retrieved confusion bank with random counterparts closes the accuracy gap to RSD on the same heterogeneous CIFAR-100 and ImageNet pairs, the calibration claim fails.","supporting_citations":[],"review_version":1}