Pith. sign in

REVIEW 2 major objections 1 minor 52 references

Sharing LoRA adapters between teacher and student encoders during joint training corrects feature misalignment and raises accuracy for both models in task-specific distillation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-29 08:29 UTC pith:RYYSVDRF

load-bearing objection SLAD shares LoRA adapters between teacher and student to improve alignment and cut training time, but the gains aren't cleanly separated from joint-training effects. the 2 major comments →

arxiv 2605.29726 v1 pith:RYYSVDRF submitted 2026-05-28 cs.CV

SLAD : Shared LoRA Adapters for Task Specific Distillation

Reda Bensaid , Yassir Bendou , Vincent Gripon , Fran\c{c}ois Leduc-Primeau This is my paper
classification cs.CV
keywords task-specific distillationLoRA adaptersfeature alignmentknowledge distillationmodel compressioncomputer visionshared parametersfoundation model adaptation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines task-specific distillation, where a larger teacher model and smaller student model of the same foundation architecture are both adapted to one downstream task to transfer knowledge from teacher to student. It identifies that fine-tuning the teacher improves its own accuracy but creates a feature representation mismatch with the student that harms distillation, whereas linear probing avoids the mismatch at the cost of lower teacher performance. The authors first apply low-rank adaptation to the teacher, then introduce a parameter-sharing strategy so that the same adapter weights are used in both models throughout joint training. This shared-adapter approach produces stronger alignment between the two feature spaces, yielding accuracy gains for the student, additional gains for the teacher, and roughly twice the training speed of full fine-tuning. Experiments across classification and segmentation benchmarks show the method reaches state-of-the-art results within the task-specific distillation setting.

Core claim

The central claim is that a parameter-sharing strategy of LoRA adapters between the teacher and student encoders during joint training corrects the feature misalignment that occurs when fine-tuning the teacher separately, resulting in improved knowledge transfer, higher accuracy for both models, and twice the training speed of fine-tuning.

What carries the argument

Shared LoRA adapters: low-rank adaptation modules whose parameters are tied between the teacher and student models during the joint training phase, enforcing feature-space alignment while keeping the number of trainable parameters small.

Load-bearing premise

The performance gap between probing and fine-tuning the teacher is caused primarily by feature misalignment during fine-tuning, and that sharing adapters will correct this misalignment without introducing offsetting drawbacks in optimization or capacity.

What would settle it

A controlled measurement of feature-space distance (for example, average cosine similarity between corresponding layer activations) on the same downstream data before and after applying the shared-adapter procedure; if alignment does not increase yet distillation performance still rises, or if alignment increases but performance does not, the misalignment-correction account would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Better feature alignment produces higher student accuracy than either separate fine-tuning or probing of the teacher.
  • The teacher model itself reaches higher final accuracy than when adapted alone.
  • Training time is halved relative to standard fine-tuning of the teacher.
  • The gains hold across both image classification and semantic segmentation datasets.
  • The approach establishes new state-of-the-art numbers inside the task-specific distillation protocol.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The result implies that explicit alignment constraints between models of different capacity can be more effective than separate optimization followed by distillation.
  • Similar parameter-tying could be tested on other adaptation techniques such as prompt tuning or prefix tuning to see whether the alignment benefit generalizes.
  • The method may reduce the total compute needed for deploying compressed models on edge devices by collapsing the usual two-stage teacher-then-student pipeline into one joint stage.
  • One could examine whether the shared adapters remain effective when the teacher and student differ more substantially in architecture rather than only in width or depth.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper proposes SLAD, a method that applies shared LoRA adapters between a larger teacher and smaller student model during joint training for task-specific distillation. It claims that fine-tuning induces feature misalignment (explaining why probing the teacher yields better distillation), that LoRA improves alignment, and that sharing adapters further enhances alignment to deliver gains for both models, 2x faster training than fine-tuning, and SOTA results on classification and segmentation datasets.

Significance. If the performance gains and causal mechanism hold under proper isolation, the approach could improve efficiency of adapting foundation models for resource-constrained settings by enabling joint adaptation without misalignment penalties.

major comments (2)
  1. [Abstract and experimental results section] Abstract and experimental results section: The central claim that feature misalignment during fine-tuning is the primary cause of weaker distillation (vs. probing) and that shared LoRA corrects it is not isolated from joint-training effects such as synchronized gradients or implicit regularization; no control experiments are described that hold joint training fixed while varying only the sharing of adapters.
  2. [Method section] Method section: The description of the parameter-sharing strategy lacks detail on whether the shared adapters receive combined gradients from both models or how capacity constraints are handled, which is load-bearing for the claim of 'no offsetting drawbacks in optimization or capacity'.
minor comments (1)
  1. The claim of being '2x faster to train than fine-tuning' would benefit from explicit specification of the baseline (e.g., separate vs. joint optimization steps) and any measured wall-clock or FLOPs comparison.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the thoughtful comments, which help clarify the isolation of effects and the optimization details in SLAD. We address each major point below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract and experimental results section] Abstract and experimental results section: The central claim that feature misalignment during fine-tuning is the primary cause of weaker distillation (vs. probing) and that shared LoRA corrects it is not isolated from joint-training effects such as synchronized gradients or implicit regularization; no control experiments are described that hold joint training fixed while varying only the sharing of adapters.

    Authors: We agree that a direct control holding joint training fixed while varying only adapter sharing would strengthen isolation of the sharing mechanism from other joint-training benefits. Our experiments compare separate fine-tuning plus distillation against joint training with shared LoRA, but lack an explicit joint-training baseline with non-shared adapters. We will add this control experiment (joint training with independent LoRA adapters per model) in the revised version to better support the claim. revision: yes

  2. Referee: [Method section] Method section: The description of the parameter-sharing strategy lacks detail on whether the shared adapters receive combined gradients from both models or how capacity constraints are handled, which is load-bearing for the claim of 'no offsetting drawbacks in optimization or capacity'.

    Authors: We will expand the method section to explicitly state that shared adapters are updated using the summed gradients from both the teacher and student task losses during joint optimization. On capacity, sharing keeps the total adapter parameters identical to using separate adapters of the same rank (no increase), and we will add a short discussion confirming no observed optimization drawbacks in our runs. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical method with no self-referential derivations or fitted predictions

full rationale

The paper advances an empirical proposal (LoRA + shared adapters for task-specific distillation) and supports it via accuracy comparisons on classification/segmentation datasets. No equations, parameter fits, or first-principles derivations appear in the provided text; performance claims are not reduced to inputs by construction. Self-citation patterns are absent from the load-bearing steps. This matches the default non-circular case for experimental method papers.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only text supplies no explicit free parameters, axioms, or invented entities; all technical details remain unspecified.

pith-pipeline@v0.9.1-grok · 5835 in / 1001 out tokens · 23237 ms · 2026-06-29T08:29:25.764534+00:00 · methodology

0 comments
read the original abstract

In the context of resource-constrained environments such as embedded systems, adapting reduced-size foundation models to downstream tasks has become increasingly popular. This has recently motivated the emerging setting of task-specific distillation, where a larger and a smaller version of the same foundation model are both adapted to the same downstream task, with the goal of transferring knowledge from the former to the latter. Recent work has demonstrated the benefits of using a larger version of the same foundation model to assist the adaptation of a smaller one. Typically, the larger model (teacher) is first adapted via fine-tuning or linear probing before its knowledge is distilled into the smaller model (student). While fine-tuning the teacher often increases its performance, recent work showed that probing it leads to better knowledge distillation to the student. Our findings show that this is mainly due to a mis-alignment in feature representation between the teacher and the student which occurs during the teacher's fine-tuning. Inspired by existing efforts to preserve previously learned knowledge, we first propose leveraging low-rank adaptation, resulting in better feature alignment and therefore better knowledge transfer. Drawing from this insight, we further enhance the feature alignment through a parameter-sharing strategy of the adapters between the two encoders during joint training. Our proposed method, SLAD, shows better feature alignment between the teacher and student, which results in increased performance for not only the student but also the teacher model, while being 2x faster to train than fine-tuning. Through extensive experiments on multiple classification and segmentation datasets, we demonstrate the improved accuracy and transfer efficiency of our method, achieving state-of-the-art performance in the task-specific distillation framework.

Figures

Figures reproduced from arXiv: 2605.29726 by Fran\c{c}ois Leduc-Primeau, Reda Bensaid, Vincent Gripon, Yassir Bendou.

Figure 1
Figure 1. Figure 1: Comparison of accuracy vs. training time for different [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our SLAD method. (left) The two-step distillation introduced in Marrie et al. [34] where the teacher is first probed on the downstream task and then distilled into a student. (right) The SLAD method where the teacher and the student are trained jointly in one step while sharing their LoRA adapter weights. ations have emerged. FitNets [39], for example, incorpo￾rated intermediate supervision to … view at source ↗
Figure 3
Figure 3. Figure 3: Centered Kernel Alignment between representations of a teacher ViT-B and a student ViT-S on CUB dataset. We plot CKA [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the sharing mechanism for SLAD: the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 4 canonical work pages · 1 internal anchor

  1. [1]

    Understanding inter- mediate layers using linear classifier probes, 2017

    Guillaume Alain and Yoshua Bengio. Understanding inter- mediate layers using linear classifier probes, 2017. 4

    2017

  2. [2]

    Zhang, Hemanth Saratchandran, Cristian Rodriguez-Opazo, Anton van den Hengel, and Ehsan Abbasnejad

    Paul Albert, Frederic Z. Zhang, Hemanth Saratchandran, Cristian Rodriguez-Opazo, Anton van den Hengel, and Ehsan Abbasnejad. RandloRA: Full rank parameter-efficient fine-tuning of large models. InThe Thirteenth International Conference on Learning Representations, 2025. 8

    2025

  3. [3]

    Do deep nets really need to be deep? InAdvances in Neural Information Processing Systems

    Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? InAdvances in Neural Information Processing Systems. Curran Associates, Inc., 2014. 2

    2014

  4. [4]

    Bit- Fit: Simple parameter-efficient fine-tuning for transformer- based masked language-models

    Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bit- Fit: Simple parameter-efficient fine-tuning for transformer- based masked language-models. InProceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 1–9, Dublin, Ire- land, 2022. Association for Computational Linguistics. 3

    2022

  5. [5]

    A novel benchmark for few-shot semantic segmen- tation in the era of foundation models, 2024

    Reda Bensaid, Vincent Gripon, Franc ¸ois Leduc-Primeau, Lukas Mauch, Ghouthi Boukli Hacene, and Fabien Car- dinaux. A novel benchmark for few-shot semantic segmen- tation in the era of foundation models, 2024. 2

    2024

  6. [6]

    Knowledge dis- tillation: A good teacher is patient and consistent

    Lucas Beyer, Xiaohua Zhai, Amelie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge dis- tillation: A good teacher is patient and consistent . In 2022 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 10915–10924, Los Alami- tos, CA, USA, 2022. IEEE Computer Society. 6

    2022

  7. [7]

    Adaptformer: Adapting vi- sion transformers for scalable visual recognition.arXiv preprint arXiv:2205.13535, 2022

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapt- ing vision transformers for scalable visual recognition.arXiv preprint arXiv:2205.13535, 2022. 4

    work page arXiv 2022

  8. [8]

    Cimpoi, S

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild. InProceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014. 6

    2014

  9. [9]

    The cityscapes dataset

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Scharw¨achter, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset. InCVPR Workshop on The Future of Datasets in Vision, 2015. 6

    2015

  10. [10]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. InProc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 6

    2016

  11. [11]

    On the effective- ness of layernorm tuning for continual learning in vision transformers

    Thomas De Min, Massimiliano Mancini, Karteek Alahari, Xavier Alameda-Pineda, and Elisa Ricci. On the effective- ness of layernorm tuning for continual learning in vision transformers. In2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 3577–3586. IEEE, 2023. 3

    2023

  12. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representa- tions, 2021. 6

    2021

  13. [13]

    Eva-02: A visual representation for neon genesis.Image and Vision Computing, 149:105171,

    Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xin- long Wang, and Yue Cao. Eva-02: A visual representation for neon genesis.Image and Vision Computing, 149:105171,

  14. [14]

    Maybank, and Dacheng Tao

    Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey.Interna- tional Journal of Computer Vision, 129(6):1789–1819, 2021. 2

    2021

  15. [15]

    Collaborative knowl- edge distillation via multiknowledge transfer.IEEE Trans- actions on Neural Networks and Learning Systems, 35(5): 6718–6730, 2024

    Jianping Gou, Liyuan Sun, Baosheng Yu, Lan Du, Kotagiri Ramamohanarao, and Dacheng Tao. Collaborative knowl- edge distillation via multiknowledge transfer.IEEE Trans- actions on Neural Networks and Learning Systems, 35(5): 6718–6730, 2024. 8

    2024

  16. [16]

    Parameter-efficient fine-tuning for large models: A comprehensive survey.Transactions on Machine Learning Research, 2024

    Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey.Transactions on Machine Learning Research, 2024. 1, 3

    2024

  17. [17]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000–16009, 2022. 1

    2022

  18. [18]

    Distilling the knowledge in a neural network, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. 1, 2

    2015

  19. [19]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InIn- ternational Conference on Learning Representations, 2022. 2, 4

    2022

  20. [20]

    Generic-to-specific distillation of masked autoencoders.IEEE Transactions on Circuits and Systems for Video Technology, 34(9):8779–8793, 2024

    Wei Huang, Zhiliang Peng, Li Dong, Furu Wei, Qixiang Ye, and Jianbin Jiao. Generic-to-specific distillation of masked autoencoders.IEEE Transactions on Circuits and Systems for Video Technology, 34(9):8779–8793, 2024. 1, 3

    2024

  21. [21]

    Like what you like: Knowl- edge distill via neuron selectivity transfer, 2017

    Zehao Huang and Naiyan Wang. Like what you like: Knowl- edge distill via neuron selectivity transfer, 2017. 3

    2017

  22. [22]

    Vi- sual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. InEuropean Conference on Computer Vision (ECCV), 2022. 3

    2022

  23. [23]

    Vi- sual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. InEuropean Conference on Computer Vision (ECCV), 2022. 11

    2022

  24. [24]

    Tinybert: Dis- tilling bert for natural language understanding, 2020

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Dis- tilling bert for natural language understanding, 2020. 1, 3, 5, 8

    2020

  25. [25]

    How to benchmark vision foundation models for semantic seg- mentation? InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Work- shops, 2024

    Tommie Kerssies, Daan de Geus, and Gijs Dubbelman. How to benchmark vision foundation models for semantic seg- mentation? InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Work- shops, 2024. 2

    2024

  26. [26]

    How to adapt your large-scale vision-and-language model, 2022

    Konwoo Kim, Michael Laskin, Igor Mordatch, and Deepak Pathak. How to adapt your large-scale vision-and-language model, 2022. 3

    2022

  27. [27]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Ku- maran, and Raia Hadsell

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Ku- maran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks.Proceedings of the National Academy of Sciences, 114(13):...

    2017

  28. [28]

    VeRA: Vector-based random matrix adaptation

    Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Represen- tations, 2024. 4

    2024

  29. [29]

    Similarity of neural network represen- tations revisited

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network represen- tations revisited. InProceedings of the 36th International Conference on Machine Learning, pages 3519–3529. PMLR,

  30. [30]

    Kullback and R

    S. Kullback and R. A. Leibler. On Information and Suffi- ciency.The Annals of Mathematical Statistics, 22(1):79 – 86, 1951. 4

    1951

  31. [31]

    Scaling down to scale up: A guide to parameter- efficient fine-tuning, 2024

    Vladislav Lialin, Vijeta Deshpande, Xiaowei Yao, and Anna Rumshisky. Scaling down to scale up: A guide to parameter- efficient fine-tuning, 2024. 1, 3

    2024

  32. [32]

    Inflora: Interference-free low-rank adaptation for continual learning

    Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23638–23647, 2024. 4

    2024

  33. [33]

    S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical re- port, 2013. 6

    2013

  34. [34]

    On good practices for task-specific distillation of large pretrained visual models.Transactions on Machine Learning Research, 2024

    Juliette Marrie, Michael Arbel, Julien Mairal, and Diane Lar- lus. On good practices for task-specific distillation of large pretrained visual models.Transactions on Machine Learning Research, 2024. 1, 2, 3, 4, 6, 7

    2024

  35. [35]

    Improved Knowledge Distillation via Teacher Assistant

    Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant: Bridging the gap between student and teacher.arXiv preprint arXiv:1902.03393, 2019. 3

    work page Pith review arXiv 1902

  36. [36]

    Maxime Oquab, Timoth ´ee Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Rus- sell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nico- las Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patri...

    2023

  37. [37]

    Rela- tional knowledge distillation

    Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Rela- tional knowledge distillation. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3967–3976, 2019. 3

    2019

  38. [38]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceedings of the 38th International Conference on Machine Learning, pages 8748–8763. PMLR, 2021. 1

    2021

  39. [39]

    Fitnets: Hints for thin deep nets

    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. InIn Proceedings of ICLR, 2015. 3

    2015

  40. [40]

    Singular value fine-tuning: Few- shot segmentation requires few-parameters fine-tuning

    Yanpeng Sun, Qiang Chen, Xiangyu He, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Jian Cheng, Zechao Li, and Jingdong Wang. Singular value fine-tuning: Few- shot segmentation requires few-parameters fine-tuning. In Advances in Neural Information Processing Systems, 2022. 4

    2022

  41. [41]

    Singular value fine-tuning: Few-shot segmentation requires few-parameters fine-tuning

    Yanpeng Sun, Qiang Chen, Xiangyu He, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Jian Cheng, Zechao Li, and Jingdong Wang. Singular value fine-tuning: Few-shot segmentation requires few-parameters fine-tuning

  42. [42]

    C. Wah, S. Branson, P. Welinder, P. Perona, and S. Be- longie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technol- ogy, 2011. 6

    2011

  43. [43]

    Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6):3048–3068, 2022. 2

    2022

  44. [44]

    Continual learning with low rank adaptation

    Martin Wistuba, Prabhu Teja S, Lukas Balles, and Giovanni Zappella. Continual learning with low rank adaptation. In NeurIPS 2023 Workshop on Distribution Shifts: New Fron- tiers with Foundation Models, 2024. 4

    2023

  45. [45]

    Tinyvit: Fast pretraining distillation for small vision transformers

    Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. InEuropean con- ference on computer vision (ECCV), 2022. 1

    2022

  46. [46]

    DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices .IEEE Transactions on Mobile Computing, 23(05):5917– 5932, 2024

    Guanyu Xu, Zhiwei Hao, Yong Luo, Han Hu, Jianping An, and Shiwen Mao. DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices .IEEE Transactions on Mobile Computing, 23(05):5917– 5932, 2024. 1

    2024

  47. [47]

    Paying more at- tention to attention: Improving the performance of convolu- tional neural networks via attention transfer

    Sergey Zagoruyko and Nikos Komodakis. Paying more at- tention to attention: Improving the performance of convolu- tional neural networks via attention transfer. InInternational Conference on Learning Representations, 2017. 3

    2017

  48. [48]

    Faster Segment Anything: Towards Lightweight SAM for Mobile Applications

    Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mo- bile applications.arXiv preprint arXiv:2306.14289, 2023. 1

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  49. [49]

    Hospedales, and Huchuan Lu

    Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. InProceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2018. 8

    2018

  50. [50]

    Effect of LoRA Rank We evaluate the sensitivity of SLAD to the rankrof the LoRA adapters on CUB (ViT-B→ViT-S)

    Additional Ablation Studies 8.1. Effect of LoRA Rank We evaluate the sensitivity of SLAD to the rankrof the LoRA adapters on CUB (ViT-B→ViT-S). Results are re- ported in Table 4. Rankr 2 4 8 16 Accuracy (%) 90.59 90.63 90.51 90.59 Table 4. Effect of LoRA rank on performance. We observe that performance remains stable across ranks, indicating that SLAD is ...

    work page arXiv

  51. [51]

    Additional Comparisons 9.1. Comparison with PEFT Methods We compare SLAD with alternative parameter-efficient fine-tuning (PEFT) methods, including Singular Value Fine-tuning (SVF) [41] and Visual Prompt Tuning (VPT) [23], on CUB (ViT-L→ViT-S). Results are shown in Table 7. Method SVF VPT LoRA SLAD Accuracy (%) 90.09 90.61 90.9291.35 Table 7. Comparison w...

  52. [52]

    Training Efficiency Standard task-specific distillation requires sequential train- ing of teacher and student, whereas SLAD jointly trains both models in a single stage

    Additional Results 10.1. Training Efficiency Standard task-specific distillation requires sequential train- ing of teacher and student, whereas SLAD jointly trains both models in a single stage. Setting Probing LoRA SLAD ViT-B→ViT-S 10.36 7.775.08 ViT-L→ViT-S 17.51 17.4910.83 ViT-L→ViT-B 18.06 19.1712.58 Table 8. Training time (in hours) for CUB dataset. ...