Pith. sign in

REVIEW 3 major objections 6 minor 44 references

TuneComp: Joint Fine-tuning and Compression for Large Foundation Models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Joint fine-tuning and compression in one pass beats sequential pipelines at the same parameter budget.

desk verdict A credible incremental extension of PC-LoRA with three concrete changes, but the 'power-preserving' annealing is mathematically mis-motivated and the empirical evidence is thin. read the letter →

arxiv 2505.21835 v1 pith:2FLMUJRU submitted 2025-05-27 cs.LG cs.AI

Xiangyu Chen , Jing Liu , Ye Wang , Matthew Brand , Pu (Perry) Wang , Toshiaki Koike-Akino This is my paper
classification cs.LGcs.AI
keywords TuneCompjointfine-tuningandcompressionknowledgedistillationlow-rankapproximationpruningactivation-awareinitializationvisiontransformersprogressive
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that fine-tuning and compression of a large pretrained model should not be done in sequence, because sequential pipelines lose accuracy and waste an intermediate large model. Instead, TuneComp carries out both at once: a frozen pretrained teacher branch and a trainable low-rank student branch run in parallel, and the student gradually replaces the teacher during training. On the ViT-Base to CIFAR-100 transfer task, the resulting accuracy-versus-parameter-size front dominates fine-tune-then-distill, distill-then-fine-tune, and progressive-compression baselines. If that comparison holds, one training run can directly produce a smaller, task-adapted model at a given parameter budget.

What carries the argument

The mechanism is the power-normalized two-branch linear layer of Eq. (5): the frozen pretrained weight $W_0$ stays active early and is phased out by a sinusoidal decay $\alpha_t$ while the trainable low-rank branch $WX+b$ is phased in with matching power $\sqrt{1-\alpha_t^2}$. Supporting machinery includes RootCorDA initialization, which performs SVD on $WC$ with $C=(XX^T)^{1/2}$ so the truncated approximation minimizes the activation-weighted error; hard-shrinkage pruning, which zeroes the smallest $\rho$ fraction of entries in the low-rank factors each forward pass; and a layer-wise feature loss whose weight decays from 1 to 0.

What would settle it

Run TuneComp and the sequential baselines on the same backbone and downstream task while varying the handover schedule (for example, keeping a constant student weight, changing the decay length, or using a constant regularization weight). If the joint method's accuracy-versus-size points stop dominating the sequential pipelines, or if the gain comes only from the RootCorDA initialization rather than the joint training, the central claim is refuted.

Watch

Extended reading notes

Core claim

The central discovery is a progressive-distillation construction in which every linear layer is replaced by the normalized mixture $Y = \alpha_t(W_0X+b_0)+\sqrt{1-\alpha_t^2}(WX+b)$, with $\alpha_t$ decaying sinusoidally from 1 to 0 over 80% of training. The student branch is a low-rank approximation initialized by an activation-aware SVD that whitens activations via $C=(XX^T)^{1/2}$ (RootCorDA), and hard shrinkage prunes the low-rank matrices during training. Layer-wise feature-matching regularization is also annealed from 1 to 0. The paper reports that this jointly trained, pruned low-rank student achieves better accuracy at each parameter budget than the sequential alternatives, and that moderate pruning (20–40%) improves the accuracy-efficiency trade-off.

Load-bearing premise

The method works only if the gradual handover from the frozen teacher to the trainable student proceeds smoothly enough that the model never loses the task knowledge it is learning; the handover schedule and normalization are chosen by design and not proven to be safe.

Editorial extensions

If this is right

  • If TuneComp's front dominates sequential compression, practitioners can replace two-stage pipelines with a single training run that yields the final deployed model.
  • Because moderate pruning (20–40%) improves the trade-off, the method can reach higher compression ratios without sacrificing accuracy relative to unpruned low-rank compression.
  • The activation-aware RootCorDA initialization alone outperforms plain SVD and CorDA initializations at low rank in the reported setting, making initialization a reusable ingredient for low-rank compression.
  • The dynamic regularization schedule improves accuracy over a constant feature-matching weight at every tested rank, suggesting early strong layer-wise guidance followed by relaxation helps joint fine-tuning and compression.
  • The reported gains are largest at high compression rates, so TuneComp is most valuable when deploying to small parameter budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension, not tested in the paper, is carrying the power-normalized teacher-to-student mixing into decoder-only language models, where progressive distillation could jointly compress attention and MLP projections during instruction tuning.
  • Because RootCorDA minimizes a weighted low-rank error using activation statistics, it may also improve other low-rank compression pipelines that currently initialize with plain SVD, even without joint fine-tuning.
  • The paper compares parameter counts but not wall-clock training time or energy; a direct measurement of total compute would show whether joint compression also reduces the cost of the whole pipeline.
  • The sinusoidal power-normalized schedule could serve as a general interpolation scheme for progressive knowledge distillation beyond low-rank students, whenever a teacher must hand off to a smaller model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. TuneComp proposes a joint fine-tuning and compression pipeline for large foundation models. Each linear layer is split into a frozen pretrained teacher branch and a trainable low-rank student branch that is also pruned. During training, the teacher output is gradually annealed into the student output via Eq. (5), with a sinusoidal decay schedule (Eq. (4)), an activation-aware low-rank initialization named RootCorDA (Section 3.2), and a layer-wise feature regularization term whose weight decays (Section 3.4.2). The method is evaluated on ViT-Base transferring from ImageNet-1K to CIFAR-100, with comparisons to fine-tune-then-distill, distill-then-fine-tune, and PC-LoRA, and the paper reports Pareto fronts over rank and pruning ratio. The main claims are that joint fine-tuning and compression yields better accuracy-versus-size trade-offs than sequential pipelines and that the proposed initialization and regularization decay each improve accuracy.

Significance. If the empirical claims are reproducible, TuneComp would offer a practical single-pass alternative to sequential fine-tuning and compression, producing a task-adapted compressed model directly with a better accuracy/size trade-off. The paper also contains a useful comparative study of low-rank initialization strategies, including the RootCorDA variant, and the idea of combining distillation, pruning, and low-rank adaptation in one loop is of clear interest to the model-compression community. However, the current evidence is thin: all results are single-run point estimates, the evaluation covers only one model and one downstream dataset, no code is released, and the theoretical motivation for the core annealing step in Eq. (5) is questionable. These issues prevent the paper from being accepted in its present form, but they are addressable.

major comments (3)
  1. [Section 3.4.1 (Eq. 5)] The stated justification for replacing PC-LoRA's alpha'=1 with alpha'_t = sqrt(1-alpha_t^2) is that this 'maintains the total power from both branches equal to 1.' This is only true if the teacher and student outputs are uncorrelated. Because the student is deliberately initialized as a low-rank approximation of the teacher (RootCorDA in Section 3.2), the outputs are strongly positively correlated, so Var(alpha Ft + sqrt(1-alpha^2) Fs) = alpha^2 Var(Ft) + (1-alpha^2) Var(Fs) + 2 alpha sqrt(1-alpha^2) Cov(Fs,Ft), which exceeds Var(Ft) whenever the covariance is positive; with perfectly correlated equal-variance outputs at alpha=1/sqrt(2), the output variance doubles. The paper provides no analysis of how this transient activation-scale inflation interacts with LayerNorm or residual streams, nor any ablation of the schedule. Since the progressive handoff is presented as the core mechanism, this is a load-bearing gap: the motivation for Eq. (5) is incorrect as stated, and the reported gains could be attributable to other components (initialization, regularization decay, pruning) rather than to the power-normalized mixing. Please correct the derivation, replace Eq. (5) with a schedule that provably preserves activation statistics under correlated branches, or show by ablation that the choice of alpha' does not affect the reported Pareto dominance.
  2. [Section 4 (Tables 1-2, Figs. 2-3)] All reported accuracies appear to be single-run point estimates with no error bars, no multiple seeds, and no significance tests. The margin between the proposed RootCorDA initialization and the best SVD-based variant is small at r=32 (78.99 vs. 78.74), and the Pareto analysis in Fig. 2 rests on these point estimates. With only one model (ViT-Base) and one downstream dataset (CIFAR-100), the claim that TuneComp 'significantly outperforms' sequential compression methods is not statistically supported. Please report mean and standard deviation over at least 3-5 seeds, provide a significance test or effect-size statement for the key comparisons in Fig. 2, and ideally add at least one more model or target dataset to demonstrate generality.
  3. [Section 3.4.1 (Eq. 4 and T=80%)] The sinusoidal decay schedule of Eq. (4) and the choice of T=80% of total iterations are introduced without any sensitivity analysis. The paper credits the joint pipeline's success to the smooth teacher-to-student handoff, but the reader cannot tell whether this exact schedule is critical or whether a simpler linear or exponential decay would work equally well. Please add an ablation over the decay horizon T and at least one alternative schedule (e.g., linear or exponential decay) for a fixed rank and pruning setting, so that the contribution of the annealing schedule itself can be assessed.
minor comments (6)
  1. [Section 3.2 (RootCorDA)] The statement that RootCorDA 'achieves the global optimal of Eq. (3) given W_hat = BA' is asserted without proof. Although it follows from the SVD-LLM derivation combined with the identity ||M C G||_F = ||M C||_F when G G^T = I, the authors should either provide a one-line proof or explicitly cite the theorem from [38] so readers can verify the claim.
  2. [Section 3.3] The hard-shrinkage pruning step is under-specified: the text states that elements below a magnitude threshold are zeroed in the forward pass, but does not state how gradients are handled for the zeroed entries or how often the pruning mask is recomputed. This is important for reproducibility, since different mask-update schemes can lead to very different optimization behavior.
  3. [Section 3.4.2] The decay of the regularization weight gamma is described only as 'decrease gamma from 1 to 0 during the iterations, similar to Eq. (4).' The exact schedule (e.g., the same sinusoidal shape and the same T=80% horizon) should be stated explicitly.
  4. [Section 4.2] The text reports that the Distill-only baseline has accuracy below 40% and omits it from Fig. 2. Please give the concrete value or a small table entry for this baseline so that the comparison is complete.
  5. [Figure 2] The caption contains the phrase '3.5x efficiency improve 10.5%' with no definition of the axes or the meaning of these numbers. Please clarify what is measured and how the efficiency improvement is computed.
  6. [General] No code or configuration details are released. Given that the paper introduces a multi-component pipeline, releasing code or at least detailed hyperparameter settings for all baselines would substantially strengthen reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central accuracy-versus-size claim is an empirical comparison against external baselines on CIFAR-100, and no fitted quantity is relabeled as a prediction.

full rationale

The paper's central claim is the Pareto-front comparison in Fig. 2 between TuneComp and sequential compression pipelines, measured by fine-tuning and testing on the external CIFAR-100 benchmark. The method's components (Eq. 4 alpha schedule, Eq. 5 power-normalized interpolation, Eq. 6 regularization, RootCorDA initialization, and hard-shrinkage pruning) are introduced with explicit equations and evaluated independently, rather than inherited from the authors' prior results. The authors' own prior works appear only as related-work citations or as an experimental-settings reference ([6] for the ViT-Base setup), neither of which is load-bearing for the accuracy claim. The main inherited baseline, PC-LoRA [18], is an external paper, and Table 2 explicitly compares against PC-LoRA's constant regularization rather than relying on it. No fitted parameter is presented as a prediction: ranks and pruning ratios are swept and reported as trade-off curves, and the learning-rate schedule is stated rather than tuned to the test set. The possible statistical correlation between teacher and student branches in Eq. (5) is a correctness or convergence concern, not a circularity step, because the reported improvement does not reduce by construction to the method's own inputs or to a self-citation chain. Thus no circular step meets the evidentiary bar, and the honest finding is a circularity score of 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces several hand-chosen schedules and compression targets that are swept or fixed without theoretical guidance. These are not fitted constants, but they are free choices on which the reported Pareto front depends. The two ad hoc assumptions listed are the least externally supported pieces of the training recipe.

free parameters (5)
  • low-rank rank r = 32, 64, 128, 256
    Chosen by hand as the compression target; smaller r gives higher compression and lower accuracy, so the reported trade-off depends on this sweep.
  • pruning ratio rho = 0%, 20%, 40%, 60%, 80%, 90%, 95%
    Chosen by hand; 20% to 40% is reported as the best accuracy-efficiency trade-off, so the claimed benefit of pruning depends on this manual sweep.
  • decay horizon T = 80% of total iterations
    Sets how many iterations the teacher branch remains active; no ablation for other fractions is reported.
  • regularization weight gamma schedule = decays from 1 to 0
    The dynamic schedule is asserted to beat the constant gamma=0.2 from PC-LoRA, but the exact functional form is not fully specified.
  • teacher-student mixing schedule alpha_t = alpha_t = 1 - sin(pi t / (2T)) for t <= T, else 0
    Schedule inherited from PC-LoRA with alpha' changed to sqrt(1 - alpha_t^2); no derivation of optimality is given.
assumptions (4)
  • standard math SVD truncation gives the optimal low-rank approximation in Frobenius norm (Eckart-Young).
    Used in Section 3.1 to define the low-rank student branch and the RootCorDA initialization.
  • domain assumption Calibration activations X are representative of downstream task activations.
    Section 3.2 computes C=(XX^T)^(1/2) from a calibration set; if the calibration distribution shifts, the activation-aware initialization and layer-wise guidance lose their stated optimality.
  • ad hoc to paper The power-normalized mixing in Eq. (5) with alpha'_t = sqrt(1 - alpha_t^2) yields stable progressive distillation.
    Introduced without derivation or ablation; it is the main change from PC-LoRA's alpha' = 1.
  • ad hoc to paper Hard shrinkage pruning of low-rank matrices during gradient training is compatible with task-loss optimization.
    Section 3.3 zeros the smallest entries each forward pass; no analysis is provided of how this interacts with gradients or rank maintenance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TuneComp: Joint Fine-tuning and Compression for Large Foundation Models." pith.science (2026). https://pith.science/paper/2FLMUJRU

@misc{pith2026250521835,
  author       = {Pith},
  title        = {Pith review of: TuneComp: Joint Fine-tuning and Compression for Large Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2FLMUJRU}},
  note         = {Machine review of arXiv:2505.21835}
}
read the original abstract

To reduce model size during post-training, compression methods, including knowledge distillation, low-rank approximation, and pruning, are often applied after fine-tuning the model. However, sequential fine-tuning and compression sacrifices performance, while creating a larger than necessary model as an intermediate step. In this work, we aim to reduce this gap, by directly constructing a smaller model while guided by the downstream task. We propose to jointly fine-tune and compress the model by gradually distilling it to a pruned low-rank structure. Experiments demonstrate that joint fine-tuning and compression significantly outperforms other sequential compression methods.

Figures

Figures reproduced from arXiv: 2505.21835 by the authors.

Figure 1
Figure 1. The proposed joint fine-tuning and compression [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison between our proposed TuneComp, jointly [PITH_FULL_IMAGE:figures/full_fig_p001_2.png] view at source ↗
Figure 3
Figure 3. Performance of TuneComp under different pruning ratio [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 36 canonical work pages

  1. [1]

    Sparsellm: Towards global pruning of pre-trained lan- guage models

    Guangji Bai, Yijiang Li, Chen Ling, Kibaek Kim, and Liang Zhao. Sparsellm: Towards global pruning of pre-trained lan- guage models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 2

    work page 2024

  2. [2]

    Post train- ing 4-bit quantization of convolutional networks for rapid- deployment

    Ron Banner, Yury Nahshan, and Daniel Soudry. Post train- ing 4-bit quantization of convolutional networks for rapid- deployment. Advances in Neural Information Processing Systems, 32, 2019. 2

    work page 2019

  3. [3]

    Zeroq: A novel zero shot quantization framework

    Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13169–13178, 2020. 2

    work page 2020

  4. [4]

    Scatterbrain: Unifying sparse and low- rank attention

    Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher R´e. Scatterbrain: Unifying sparse and low- rank attention. Advances in Neural Information Processing Systems, 34:17413–17426, 2021. 2

    work page 2021

  5. [5]

    Comprehensive survey of model compression and speed up for vision transformers.Journal of Information, Technology and Policy, pages 1–12, 2024

    Feiyang Chen, Ziqian Luo, Lisang Zhou, Xueting Pan, and Ying Jiang. Comprehensive survey of model compression and speed up for vision transformers.Journal of Information, Technology and Policy, pages 1–12, 2024. 2

    work page 2024

  6. [6]

    Super- LoRA: Parameter-efficient unified adaptation for large vi- sion models

    Xiangyu Chen, Jing Liu, Ye Wang, Pu Wang, Matthew Brand, Guanghui Wang, and Toshiaki Koike-Akino. Super- LoRA: Parameter-efficient unified adaptation for large vi- sion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Work- shops, pages 8050–8055, 2024. 2, 3

    work page 2024

  7. [7]

    Slaying the hydra: Parameter-efficient hyper networks with low-displacement rank adaptation

    Xiangyu Chen, Ye Wang, Matthew Brand, Pu Perry Wang, Jing Liu, and Toshiaki Koike-Akino. Slaying the hydra: Parameter-efficient hyper networks with low-displacement rank adaptation. In Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning, 2024. 2

    work page 2024

  8. [8]

    A survey on deep neural network pruning: Taxonomy, compar- ison, analysis, and recommendations

    Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A survey on deep neural network pruning: Taxonomy, compar- ison, analysis, and recommendations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 2

    work page 2024

Show all 44 references
  1. [9]

    Rethinking attention with performers

    Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sar- los, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations ,

  2. [10]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems, 2022. 2

    2022

  3. [11]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. InInternational ...

    2020

  4. [12]

    KronA: Parameter efficient tuning with Kronecker adapter

    Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. KronA: Parameter efficient tuning with Kronecker adapter. In NeurIPS’23 Workshop on on Efficient Natural Language and Speech Processing, 2023. 2

    2023

  5. [13]

    Sparsegpt: massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: massive language models can be accurately pruned in one-shot. InProceedings of the 40th International Conference on Machine Learning , pages 10323–10337, 2023. 2

    2023

  6. [14]

    The impact of initialization on lora finetuning dynamics

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. The impact of initialization on lora finetuning dynamics. arXiv preprint arXiv:2406.08447, 2024. 2, 3, 4

    2024 arXiv

  7. [15]

    Language model compression with weighted low-rank factorization

    Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In International Confer- ence on Learning Representations, 2022. 2

    2022

  8. [16]

    Lora: Low- rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low- rank adaptation of large language models. In International Conference on Learning Representations, 2022. 2

    2022

  9. [17]

    Com- pressing speaker extraction model with ultra-low precision quantization and knowledge distillation

    Yating Huang, Yunzhe Hao, Jiaming Xu, and Bo Xu. Com- pressing speaker extraction model with ultra-low precision quantization and knowledge distillation. Neural Networks, 154:13–21, 2022. 2

    2022

  10. [18]

    PC-loRA: Low-rank adaptation for progressive model compression with knowledge distilla- tion

    Injoon Hwang, Haewon Park, Youngwan Lee, Jooyoung Yang, and SunJae Maeng. PC-loRA: Low-rank adaptation for progressive model compression with knowledge distilla- tion. arXiv preprint arXiv:2406.09117, 2024. 1, 2, 3, 4

    2024 arXiv

  11. [19]

    Gpt-zip: Deep compres- sion of finetuned large language models

    Berivan Isik, Hermann Kumbong, Wanyi Ning, Xiaozhe Yao, Sanmi Koyejo, and Ce Zhang. Gpt-zip: Deep compres- sion of finetuned large language models. In Workshop on Ef- ficient Systems for Foundation Models@ ICML2023 , 2023. 2

    2023

  12. [20]

    A neural network com- pression method based on knowledge-distillation and param- eter quantization for the bearing fault diagnosis.Applied Soft Computing, 127:109331, 2022

    Mengyu Ji, Gaoliang Peng, Sijue Li, Feng Cheng, Zhao Chen, Zhixiong Li, and Haiping Du. A neural network com- pression method based on knowledge-distillation and param- eter quantization for the bearing fault diagnosis.Applied Soft Computing, 127:109331, 2022. 2

    2022

  13. [21]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 3

    2009

  14. [22]

    Reward design with language models

    Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In The Eleventh International Conference on Learning Representa- tions, 2023. 2

    2023

  15. [23]

    A fast post- training pruning framework for transformers

    Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post- training pruning framework for transformers. Advances in Neural Information Processing Systems , 35:24101–24116,

  16. [24]

    On the crucial role of initialization for matrix factorization

    Bingcong Li, Liang Zhang, Aryan Mokhtari, and Niao He. On the crucial role of initialization for matrix factorization. arXiv preprint arXiv:2410.18965, 2024. 2, 4

    2024 arXiv

  17. [25]

    Yolo-based face mask detection on low-end devices using pruning and quantization

    Benedetta Liberatori, Ciro Antonio Mami, Giovanni San- tacatterina, Marco Zullich, and Felice Andrea Pellegrino. Yolo-based face mask detection on low-end devices using pruning and quantization. In 2022 45th Jubilee International Convention on Information, Communication and El...

    2022

  18. [26]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, 5 Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems , 6:87–100,

  19. [27]

    Loda: Low-dimensional adaptation of large language models

    Jing Liu, Toshiaki Koike-Akino, Pu Wang, Matthew Brand, Ye Wang, and Kieran Parsons. Loda: Low-dimensional adaptation of large language models. In NeurIPS’23 Work- shop on on Efficient Natural Language and Speech Process- ing, 2023. 2

    2023

  20. [28]

    Rethinking the value of network pruning

    Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations ,

  21. [29]

    Com- puter vision model compression techniques for embedded systems: A survey

    Alexandre Lopes, Fernando Pereira dos Santos, Diulhio de Oliveira, Mauricio Schiezaro, and Helio Pedrini. Com- puter vision model compression techniques for embedded systems: A survey. Computers & Graphics , 123:104015,

  22. [30]

    Shortgpt: Layers in large language models are more redun- dant than you expect

    Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redun- dant than you expect. arXiv preprint arXiv:2403.03853 ,

    arXiv

  23. [31]

    PiSSA: Principal singular values and singular vectors adaptation of large language models

    Fanxu Meng, Zhaohui Wang, and Muhan Zhang. PiSSA: Principal singular values and singular vectors adaptation of large language models. In The Thirty-eighth Annual Confer- ence on Neural Information Processing Systems, 2024. 2

    2024

  24. [32]

    Data-free quantization through weight equal- ization and bias correction

    Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equal- ization and bias correction. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1325– 1334, 2019. 2

    2019

  25. [33]

    Structured unrestricted-rank matrices for parameter efficient finetuning

    Arijit Sehanobish, Kumar Avinava Dubey, Krzysztof Marcin Choromanski, Somnath Basu Roy Chowdhury, Deepali Jain, Vikas Sindhwani, and Snigdha Chaturvedi. Structured unrestricted-rank matrices for parameter efficient finetuning. In The Thirty-eighth Annual Conference on Neural I...

    2024

  26. [34]

    Sanity-checking prun- ing methods: Random tickets can win the jackpot

    Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao, Liwei Wang, and Jason D Lee. Sanity-checking prun- ing methods: Random tickets can win the jackpot. Advances in neural information processing systems, 33:20390–20401,

  27. [35]

    Fine-pruning: Joint fine-tuning and compression of a convo- lutional network with bayesian optimization

    Frederick Tung, Srikanth Muralidharan, and Greg Mori. Fine-pruning: Joint fine-tuning and compression of a convo- lutional network with bayesian optimization. arXiv preprint arXiv:1707.09102, 2017. 1, 2

    2017 arXiv

  28. [36]

    Gan slimming: All-in-one gan compres- sion by a unified optimization framework

    Haotao Wang, Shupeng Gui, Haichuan Yang, Ji Liu, and Zhangyang Wang. Gan slimming: All-in-one gan compres- sion by a unified optimization framework. InEuropean Con- ference on Computer Vision , pages 54–73. Springer, 2020. 2

    2020

  29. [37]

    Pufferfish: Communication-efficient models at no extra cost

    Hongyi Wang, Saurabh Agarwal, and Dimitris Papailiopou- los. Pufferfish: Communication-efficient models at no extra cost. Proceedings of Machine Learning and Systems, 3:365– 386, 2021. 2

    2021

  30. [38]

    SVD-LLM: Truncation-aware singular value decomposition for large language model compression

    Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378, 2024. 2, 3

    2024 arXiv

  31. [39]

    Nystr¨omformer: A nystr¨om-based algorithm for approximat- ing self-attention

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystr¨omformer: A nystr¨om-based algorithm for approximat- ing self-attention. In Proceedings of the AAAI conference on artificial intelligence, pages 14138–14148, 2021. 2

    2021

  32. [40]

    CorDA: Context-oriented decomposition adaptation of large language models for task-aware parameter-efficient fine- tuning

    Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. CorDA: Context-oriented decomposition adaptation of large language models for task-aware parameter-efficient fine- tuning. In The Thirty-eighth Annual Conference on Neural I...

    2024

  33. [41]

    Joint-detnas: Upgrade your detector with nas, pruning and dynamic distillation

    Lewei Yao, Renjie Pi, Hang Xu, Wei Zhang, Zhenguo Li, and Tong Zhang. Joint-detnas: Upgrade your detector with nas, pruning and dynamic distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10175–10184, 2021. 2

    2021

  34. [42]

    Navigating text-to- image customization: From LyCORIS fine-tuning to model evaluation

    Shih-Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. Navigating text-to- image customization: From LyCORIS fine-tuning to model evaluation. In The Twelfth International Conference on Learning Representations, 2024. 2

    2024

  35. [43]

    Rptq: Reorder-based post-training quantization for large language models

    Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01089, 2023. 2

    2023 arXiv

  36. [44]

    ASVD: Activation-aware singular value decomposition for compressing large language models

    Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. ASVD: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023. 2 6

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.