Pith. sign in

REVIEW 4 cited by

LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.18356 v2 pith:HDIRBNKB submitted 2023-10-24 cs.CL cs.AIcs.LG

Tianyi Chen , Tianyu Ding , Badal Yadav , Ilya Zharkov , Luming Liang This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords lorashearknowledgellmspruningadaptorsdynamiceffectivelyefficient
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) have transformed the landscape of artificial intelligence, while their enormous size presents significant challenges in terms of computational costs. We introduce LoRAShear, a novel efficient approach to structurally prune LLMs and recover knowledge. Given general LLMs, LoRAShear at first creates the dependency graphs over LoRA modules to discover minimally removal structures and analyze the knowledge distribution. It then proceeds progressive structured pruning on LoRA adaptors and enables inherent knowledge transfer to better preserve the information in the redundant structures. To recover the lost knowledge during pruning, LoRAShear meticulously studies and proposes a dynamic fine-tuning schemes with dynamic data adaptors to effectively narrow down the performance gap to the full models. Numerical results demonstrate that by only using one GPU within a couple of GPU days, LoRAShear effectively reduced footprint of LLMs by 20% with only 1.0% performance degradation and significantly outperforms state-of-the-arts. The source code will be available at https://github.com/microsoft/lorashear.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation

    cs.LG 2025-09 conditional novelty 6.0 of 10

    PrunedLoRA tunes large-rank LoRA adapters and progressively structurally prunes A and B to a target low rank, empirically closing much of the LoRA-to-full-fine-tuning gap.

  2. EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Pruning-aware pretraining iteratively deletes low-saliency parameter groups from a larger pretrained LLM during continued training, yielding sub-1B EfficientLLM models that outperform direct-pretraining baselines.

  3. SlimLLM: Accurate Structured Pruning for Large Language Models

    cs.LG 2025-05 conditional novelty 5.0 of 10

    SlimLLM prunes LLMs by scoring whole attention heads and feed-forward channels with Pearson similarity and PCA-based measures, then restores accuracy with per-dimension linear regression and non-uniform layer ratios.

  4. Model Merging for Knowledge Editing

    cs.AI 2025-06 reject novelty 4.0 of 10

    R-SFT plus task-vector scaling and pruning is proposed for knowledge editing, but the claimed sequential-editing advantage is not validated by the reported experiments.

Pith tools