pith. sign in

arxiv: 2606.24557 · v1 · pith:G5IK6WIPnew · submitted 2026-06-23 · 💻 cs.CV

Heterogeneous Knowledge Distillation via Geometry Decoupling and Momentum-Aware Gradient Regulation

Wuming Yang , Xiang Zhang , Hongmin Zhao This is my paper

Pith reviewed 2026-06-26 00:39 UTC · model grok-4.3

classification 💻 cs.CV
keywords Heterogeneous Knowledge DistillationKnowledge DistillationLayerNorm DecouplingGradient RegulationFeature AlignmentModel CompressionComputer VisionTraining Stability
0
0 comments X

The pith

SPOFA stabilizes heterogeneous knowledge distillation by decoupling feature geometry with LayerNorm and regulating gradients via momentum-aware scaling.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper seeks to establish a stable method for transferring knowledge between dissimilar neural network architectures, such as from transformers to convolutional networks, where direct distillation typically fails due to mismatched feature scales and opposing gradient signals. It identifies massive norm differences that slow optimization and gradient clashes from distinct model biases as the paired sources of instability. The proposed solution introduces a projector that separates feature magnitude from direction using LayerNorm, paired with a momentum-based scaler that tracks historical gradients to suppress conflicting updates. This dual approach is designed to produce reliable convergence while keeping added computation near zero. If effective, it would make cross-architecture distillation practical on standard vision benchmarks without the expense of prior stabilization techniques.

Core claim

The central claim is that heterogeneous knowledge distillation instability arises from two coupled problems—feature norm discrepancies causing optimization drag and gradient conflicts from differing inductive biases—and that these are resolved by a LayerNorm-based decoupling projector that creates a bounded space for semantic alignment plus a momentum-driven exponential moving average dynamic scaler that evaluates and penalizes harmful distillation gradients, yielding stable training with minimal overhead.

What carries the argument

The Feature and Gradient Dual Stabilization mechanism, where a LayerNorm-based decoupling projector separates feature magnitude from direction and a momentum-driven Exponential Moving Average (MEMA) dynamic scaler adaptively adjusts gradient contributions based on historical optimization trajectory.

If this is right

  • Knowledge can be transferred reliably between architectures with incompatible inductive biases without manual tuning of loss weights.
  • The added parameters and compute remain negligible compared with standard knowledge distillation baselines.
  • Performance exceeds that of prior methods that rely on heavier regularization or architectural changes.
  • Convergence occurs without the optimization drag previously associated with raw feature alignment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same decoupling step could be inserted into other alignment losses where scale mismatches appear, such as in self-supervised representation learning.
  • MEMA-style conflict detection might extend to multi-task learning settings that suffer from competing objectives.
  • Testing the method on non-vision modalities would reveal whether the norm-and-gradient pattern is architecture-specific or general.

Load-bearing premise

The observed instability in heterogeneous distillation is produced by the specific combination of feature norm mismatches and gradient conflicts that can be directly mitigated by explicit LayerNorm decoupling and momentum-based gradient scaling.

What would settle it

Running the proposed projector and scaler on transformer-to-CNN distillation tasks and observing no reduction in measured gradient conflict metrics or no improvement in final accuracy relative to the unregularized baseline would falsify the central claim.

Figures

Figures reproduced from arXiv: 2606.24557 by Hongmin Zhao, Wuming Yang, Xiang Zhang.

Figure 1
Figure 1. Figure 1: Observation of Gradient Conflicts and the SPOFA Rectification [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The overall architecture of the proposed SPOFA framework. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparative analysis of feature-level dynamics during heterogeneous [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Hyperparameter sensitivity analysis of the MEMA scaler on CIFAR [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of Teacher-Student logits cosine similarity. SPOFA [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 8
Figure 8. Figure 8: Convergence analysis of training losses over 120,000 steps. SPOFA exhibits accelerated and deeper convergence, particularly in the Distillation Loss [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
read the original abstract

Heterogeneous Knowledge Distillation (HKD) aims to transfer knowledge across varying architectures (e.g., from Transformer to CNN) but inherently suffers from severe training instability. We reveal that this instability stems from two highly coupled challenges: massive feature norm discrepancies that cause optimization drag, and severe gradient conflicts between the primary and distillation objectives arising from distinct inductive biases. To achieve stable distillation, we propose SPOFA, a framework built upon a novel Feature and Gradient Dual Stabilization mechanism. Specifically, at the feature level, we introduce a LayerNorm-based decoupling projector that explicitly decouples feature magnitude from direction, creating a bounded and stable space for semantic alignment. At the gradient level, we propose a momentum-driven Exponential Moving Average (MEMA) dynamic scaler. By establishing a robust historical baseline of the optimization trajectory, MEMA actively evaluates instantaneous gradient conflicts and adaptively penalizes harmful distillation signals, guaranteeing stable convergence. Importantly, SPOFA achieves this dual stabilization with an extremely lightweight parameter footprint. Extensive experiments on two mainstream benchmarks demonstrate that SPOFA achieves state-of-the-art accuracy, significantly outperforming computationally expensive methods while introducing only minimal computational overhead compared to standard baselines.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper claims that training instability in heterogeneous knowledge distillation (HKD) arises from coupled feature-norm discrepancies and gradient conflicts due to differing inductive biases; it proposes the SPOFA framework with a LayerNorm-based decoupling projector for feature-level stabilization and a momentum-driven exponential moving average (MEMA) dynamic scaler for gradient-level conflict penalization, asserting that this yields SOTA accuracy on two mainstream benchmarks while adding only minimal computational overhead relative to standard baselines.

Significance. If the empirical claims are substantiated, the work could offer a lightweight stabilization technique for cross-architecture distillation (e.g., Transformer-to-CNN), reducing reliance on computationally heavy alternatives and addressing a practical bottleneck in knowledge transfer for computer vision models.

major comments (3)
  1. [Abstract] Abstract: the central SOTA claim is presented without reference to any table, figure, dataset, metric, or error-bar detail, leaving the primary empirical assertion unsupported at the level of the provided text and making it impossible to assess whether the proposed mechanisms outperform baselines.
  2. [Abstract] Abstract: the LayerNorm-based decoupling projector and MEMA scaler are introduced as novel components without equations, pseudocode, or derivation showing how magnitude/direction decoupling is performed or how the historical baseline evaluates gradient conflicts; this is load-bearing for the dual-stabilization claim.
  3. [Abstract] Abstract: the momentum coefficient in MEMA is identified as a free parameter, yet the text asserts an 'extremely lightweight parameter footprint' and 'parameter-free' stabilization; the relationship between this tunable coefficient and the claimed minimal overhead is not clarified.
minor comments (1)
  1. [Abstract] Abstract: the phrase 'two mainstream benchmarks' is not named, hindering immediate assessment of generality.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments focused on the abstract. We agree that the abstract can be strengthened to better support the claims and will revise it in the next version while preserving its concise nature. All points raised can be addressed through targeted revisions to the abstract text.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central SOTA claim is presented without reference to any table, figure, dataset, metric, or error-bar detail, leaving the primary empirical assertion unsupported at the level of the provided text and making it impossible to assess whether the proposed mechanisms outperform baselines.

    Authors: We agree that the abstract should provide pointers to the supporting evidence. In the revised version, we will update the final sentence of the abstract to explicitly reference the main results: 'Extensive experiments on CIFAR-100 and ImageNet (Tables 1-3) demonstrate that SPOFA achieves state-of-the-art top-1 accuracy, significantly outperforming prior methods with standard deviations reported across three runs.' This directly links the SOTA claim to the empirical tables and metrics. revision: yes

  2. Referee: [Abstract] Abstract: the LayerNorm-based decoupling projector and MEMA scaler are introduced as novel components without equations, pseudocode, or derivation showing how magnitude/direction decoupling is performed or how the historical baseline evaluates gradient conflicts; this is load-bearing for the dual-stabilization claim.

    Authors: The abstract is a high-level summary; full equations, derivations, and pseudocode appear in Sections 3.2 (LayerNorm projector) and 3.3 (MEMA formulation). To address the concern, we will revise the abstract by adding one concise clause: 'via explicit magnitude-direction decoupling with LayerNorm and conflict-aware scaling against an EMA historical baseline.' This provides the requested intuition without exceeding abstract length limits. revision: yes

  3. Referee: [Abstract] Abstract: the momentum coefficient in MEMA is identified as a free parameter, yet the text asserts an 'extremely lightweight parameter footprint' and 'parameter-free' stabilization; the relationship between this tunable coefficient and the claimed minimal overhead is not clarified.

    Authors: The momentum coefficient is a single scalar hyperparameter (set to 0.99 in experiments), not a learnable model parameter. The 'extremely lightweight parameter footprint' and stabilization refer to the absence of additional network weights or architectural changes. We will revise the abstract to clarify: 'with an extremely lightweight parameter footprint (single scalar momentum coefficient, no extra learnable parameters).' This distinction will also be emphasized in the methods section. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper presents a descriptive proposal for SPOFA that identifies two challenges in HKD and introduces LayerNorm decoupling plus MEMA scaling as remedies. No equations, derivations, fitted parameters renamed as predictions, or self-citation chains appear in the supplied text. The central claims rest on empirical results and novel components rather than any reduction of outputs to inputs by construction, making the derivation self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 2 invented entities

Ledger extracted from abstract descriptions only; no full paper available to audit additional parameters or assumptions.

free parameters (1)
  • momentum coefficient in MEMA
    Implied as part of the dynamic scaler but value and fitting procedure not specified in abstract.
axioms (1)
  • domain assumption Instability in HKD arises specifically from coupled feature norm discrepancies and gradient conflicts between primary and distillation objectives.
    Directly stated in the abstract as the revealed cause.
invented entities (2)
  • LayerNorm-based decoupling projector no independent evidence
    purpose: Explicitly decouples feature magnitude from direction to create bounded space for semantic alignment.
    New component introduced to address feature-level instability.
  • MEMA dynamic scaler no independent evidence
    purpose: Uses historical optimization trajectory baseline to evaluate and penalize harmful distillation gradient signals.
    New component introduced to address gradient-level instability.

pith-pipeline@v0.9.1-grok · 5731 in / 1284 out tokens · 26769 ms · 2026-06-26T00:39:05.955374+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 3 canonical work pages · 1 internal anchor

  1. [1]

    Distilling the Knowledge in a Neural Network

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,”arXiv preprint arXiv:1503.02531, 2015

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  2. [2]

    Reflectance prediction- based knowledge distillation for robust 3d object detection in com- pressed point clouds,

    H. Jing, A. Wang, Y . Zhang, D. Bu, and J. Hou, “Reflectance prediction- based knowledge distillation for robust 3d object detection in com- pressed point clouds,”IEEE Transactions on Image Processing, vol. 35, pp. 85–97, 2026

    2026

  3. [3]

    One-for- all: Bridge the gap between heterogeneous architectures in knowledge distillation,

    Z. Hao, J. Guo, K. Han, Y . Tang, H. Hu, Y . Wang, and C. Xu, “One-for- all: Bridge the gap between heterogeneous architectures in knowledge distillation,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023, pp. 2826–2847

    2023

  4. [4]

    Perspective-aware teaching: Adapting knowledge for heterogeneous distillation,

    J.-H. Lin, Y . Yao, C.-F. Hsu, H. Xie, H.-H. Shuai, and W.-H. Cheng, “Perspective-aware teaching: Adapting knowledge for heterogeneous distillation,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025

    2025

  5. [5]

    Do vision transformers see like convolutional neural networks?

    M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy, “Do vision transformers see like convolutional neural networks?” in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021, pp. 12 116–12 128

    2021

  6. [6]

    Trust-calibrated collaborative learning for long- tailed visual recognition,

    H. Zhou and T. Luo, “Trust-calibrated collaborative learning for long- tailed visual recognition,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026, pp. 40 866–40 875

    2026

  7. [7]

    Paying more attention to attention: Improving the performance of convolutional neural networks via atten- tion transfer,

    S. Zagoruyko and N. Komodakis, “Paying more attention to attention: Improving the performance of convolutional neural networks via atten- tion transfer,” inInternational Conference on Learning Representations (ICLR), 2017

    2017

  8. [8]

    Fitnets: Hints for thin deep nets,

    A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y . Ben- gio, “Fitnets: Hints for thin deep nets,” inInternational Conference on Learning Representations (ICLR), 2015

    2015

  9. [9]

    Relational knowledge distilla- tion,

    W. Park, D. Kim, Y . Lu, and M. Cho, “Relational knowledge distilla- tion,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 3967–3976

    2019

  10. [10]

    Heteroge- neous generative knowledge distillation with masked image modeling,

    Z. Wang, S. Han, X. Wang, J. Hao, X. Cao, and B. Zhang, “Heteroge- neous generative knowledge distillation with masked image modeling,” arXiv preprint arXiv:2309.09571, 2023

    work page arXiv 2023

  11. [11]

    Data-free knowledge distillation for het- erogeneous federated learning,

    Z. Zhu, J. Hong, and J. Zhou, “Data-free knowledge distillation for het- erogeneous federated learning,” inInternational Conference on Machine Learning (ICML). PMLR, 2021, pp. 12 878–12 889

    2021

  12. [12]

    Heterogeneous mutual knowledge distillation for wearable human activity recognition,

    Z. Xiao, H. Xing, R. Qu, H. Li, X. Cheng, L. Xu, L. Feng, and Q. Wan, “Heterogeneous mutual knowledge distillation for wearable human activity recognition,”IEEE transactions on neural networks and learning systems, 2025

    2025

  13. [13]

    D3still: Decoupled differential distillation for asymmetric image retrieval,

    Y . Xie, Y . Lin, W. Cai, X. Xu, H. Zhang, Y . Du, and S. He, “D3still: Decoupled differential distillation for asymmetric image retrieval,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 17 181–17 190

    2024

  14. [14]

    Style blind domain generalized semantic segmentation via covariance alignment and semantic consistence contrastive learning,

    W.-J. Ahn, G.-Y . Yang, H.-D. Choi, and M.-T. Lim, “Style blind domain generalized semantic segmentation via covariance alignment and semantic consistence contrastive learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 3325–3335

    2024

  15. [15]

    Decoupled doubly contrastive learning for cross domain facial action unit detection,

    Y . Li, M. Liu, Z. Cui, Y . Ding, Y . Zong, W. Zheng, S. Shan, and C. Guan, “Decoupled doubly contrastive learning for cross domain facial action unit detection,”IEEE Transactions on Image Processing, 2025

    2025

  16. [16]

    Orthogonal decoupling contrastive regularization: Towards uncorrelated feature decoupling for unpaired image restoration,

    Z. Wang, J. Peng, H. Zhao, L. Yao, and K. Zhao, “Orthogonal decoupling contrastive regularization: Towards uncorrelated feature decoupling for unpaired image restoration,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  17. [17]

    Vkd: Improving knowledge distillation using orthogonal projections,

    R. Miles, I. Elezi, and J. Deng, “Vkd: Improving knowledge distillation using orthogonal projections,” inProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2024, pp. 15 720– 15 730

    2024

  18. [18]

    Scaled decoupled distillation,

    S. Wei, C. Luo, and Y . Luo, “Scaled decoupled distillation,” inPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 15 975–15 983

    2024

  19. [19]

    Distilling structural knowledge from cnns to vision transformers for data-efficient visual recognition,

    D. Chen, X. Teng, X. Shen, X. Yang, and L. Lan, “Distilling structural knowledge from cnns to vision transformers for data-efficient visual recognition,”Neural Networks, p. 108601, 2026

    2026

  20. [20]

    Gradient surgery for multi-task learning,

    T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn, “Gradient surgery for multi-task learning,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 5824– 5836

    2020

  21. [21]

    Conflict-averse gradient descent for multi-task learning,

    B. Liu, X. Liu, X. Jin, P. Stone, and Q. Liu, “Conflict-averse gradient descent for multi-task learning,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021, pp. 18 878–18 890

    2021

  22. [22]

    Dto-kd: Dynamic trade-off optimization for effective knowledge distillation,

    Z. Hayder, A. Cheraghian, L. Petersson, M. Harandi, and R. Hart- ley, “Dto-kd: Dynamic trade-off optimization for effective knowledge distillation,” inThe Fourteenth International Conference on Learning Representations, 2026

    2026

  23. [23]

    Dist+: Knowledge distillation from a stronger adaptive teacher,

    T. Huang, S. You, F. Wang, C. Qian, and C. Xu, “Dist+: Knowledge distillation from a stronger adaptive teacher,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  24. [24]

    Safe and balanced: A framework for constrained multi-objective reinforce- ment learning,

    S. Gu, B. Sel, Y . Ding, L. Wang, Q. Lin, A. Knoll, and M. Jin, “Safe and balanced: A framework for constrained multi-objective reinforce- ment learning,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  25. [25]

    Continuous review and timely correction: Enhancing the resistance to noisy labels via self-not- true and class-wise distillation,

    L. Lan, J. Wang, X. Wu, B. Han, and X. Liu, “Continuous review and timely correction: Enhancing the resistance to noisy labels via self-not- true and class-wise distillation,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  26. [26]

    Toward generalizable prompt learning via multi-regularization guided knowledge distillation,

    X. Yang, X. Zhong, D. Kong, and N. Wang, “Toward generalizable prompt learning via multi-regularization guided knowledge distillation,” IEEE Transactions on Image Processing, vol. 34, pp. 7600–7612, 2025

    2025

  27. [27]

    Learning to unlearn while retaining: Combat- ing gradient conflicts in machine unlearning,

    G. Patel and Q. Qiu, “Learning to unlearn while retaining: Combat- ing gradient conflicts in machine unlearning,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 4211–4221

    2025

  28. [28]

    Two losses, one goal: Balancing conflict gradients for semi-supervised semantic segmen- tation,

    R. Sun, H. Mai, W. Li, Y . Chen, and Y . Wang, “Two losses, one goal: Balancing conflict gradients for semi-supervised semantic segmen- tation,” inProceedings of the IEEE/CVF international conference on computer vision, 2025, pp. 20 357–20 367

    2025

  29. [29]

    Ckd: Contrastive knowl- edge distillation from a sample-wise perspective,

    W. Zhu, X. Zhou, P. Zhu, Y . Wang, and Q. Hu, “Ckd: Contrastive knowl- edge distillation from a sample-wise perspective,”IEEE Transactions on Image Processing, 2025

    2025

  30. [30]

    A teacher-free graph knowledge distillation framework with dual self-distillation,

    L. Wu, H. Lin, Z. Gao, G. Zhao, and S. Z. Li, “A teacher-free graph knowledge distillation framework with dual self-distillation,”IEEE Transactions on Knowledge and Data Engineering, vol. 36, no. 9, pp. 4375–4385, 2024

    2024

  31. [31]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hintonet al., “Learning multiple layers of features from tiny images,” University of Toronto, Tech. Rep., 2009

    2009

  32. [32]

    Ima- genet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Ima- genet: A large-scale hierarchical image database,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2009, pp. 248–255

    2009

  33. [33]

    Swin transformer v2: Scaling up capacity and resolution,

    Z. Liu, H. Hu, Y . Lin, Z. Yao, Z. Xie, Y . Wei, J. Ning, Y . Cao, Z. Zhang, L. Donget al., “Swin transformer v2: Scaling up capacity and resolution,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 12 009–12 019

    2022

  34. [34]

    A convnet for the 2020s,

    Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 11 976– 11 986

    2022

  35. [35]

    Transformers in vision: A survey,

    S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,”ACM Computing Surveys (CSUR), vol. 54, no. 10s, pp. 1–41, 2022

    2022

  36. [36]

    Crosskd: Cross-head knowledge distillation for object detection,

    J. Wanget al., “Crosskd: Cross-head knowledge distillation for object detection,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 1–10

    2024

  37. [37]

    Aligning logits genera- tively for principled black-box knowledge distillation,

    J. Ma, X. Xiang, K. Wang, Y . Wu, and Y . Li, “Aligning logits genera- tively for principled black-box knowledge distillation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 23 148–23 157

    2024

  38. [38]

    Decoupled knowledge distillation,

    B. Zhao, Q. Cui, R. Song, Y . Qiu, and J. Liang, “Decoupled knowledge distillation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 11 953–11 962

    2022

  39. [39]

    Knowledge distillation from a stronger teacher,

    T. Huang, S. Y . Yuan, F. Wang, C. Qian, and C. Xu, “Knowledge distillation from a stronger teacher,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, pp. 33 716–33 727

    2022

  40. [40]

    Contrastive representation distil- lation,

    Y . Tian, D. Krishnan, and P. Isola, “Contrastive representation distil- lation,” inProceedings of the International Conference on Learning Representations (ICLR), 2020

    2020

  41. [41]

    The unseen bias: How norm discrepancy in pre-norm mllms leads to visual information loss,

    B. Li, X. Xue, S. Yang, Y . Shi, X. Chen, Y . Guan, Y . Zhang, and W. Zhang, “The unseen bias: How norm discrepancy in pre-norm mllms leads to visual information loss,”arXiv preprint arXiv:2512.08374, 2025

    work page arXiv 2025