Pith. sign in

REVIEW 3 major objections 5 minor 48 references

ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ImPart claims that importance-aware SVD-space sparsification roughly doubles the compression ratio of fine-tuned LLM deltas at the same accuracy, and that this transfers to quantization and merging.

desk verdict A sensible SVD-based delta-sparsification idea with solid ablations, but the printed allocation algorithm overshoots the target sparsity, so the headline compression ratios are not anchored. read the letter →

arxiv 2504.13237 v1 pith:5MSJ6EEU submitted 2025-04-17 cs.CL

Yan Yang , Yixia Li , Hongru Wang , Xuetao Wei , Jianqiao Yu , Yun Chen , Guanhua Chen This is my paper · ORCID
classification cs.CL
keywords deltacompressionsparsificationsingularvaluedecompositionmodelmergingquantizationimportance-awarepruninglargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ImPart's claim is that the right place to sparsify a fine-tuned model's delta is the SVD domain, and the right question is which singular vector matters, not which entry is large. By assigning each singular-vector pair a drop probability that rises as its singular value falls, with a long tail pre-pruned entirely and survivors rescaled by $1/(1-p_k)$, the method keeps the reconstructed output unbiased in expectation. The reported consequence is a compression ratio about twice as high as DARE or LowRank sparsification at the same task accuracy, plus gains when the sparsified deltas are quantized or merged. If correct, ImPart would make it cheaper to store, serve, and combine many task-specific models from a shared backbone.

What carries the argument

The engine is an adaptive allocation of sparsity ratios to singular-vector pairs. After writing the delta as $\Delta W = U\Sigma V^{\top} = \sum_{k=1}^{n} \sigma_k U_k V_k^{\top}$, the method assigns pair $k$ a drop probability $p_k = (1-(\sigma_k/\sigma_1)^C)\gamma$ for the retained singular components, sets $p_k=1$ for the pre-pruned long tail beyond rank $\lfloor n(1-\beta)\rfloor$, and rescales surviving entries by $1/(1-p_k)$. Independent Bernoulli masks then randomize $U_k$ and $V_k$, and the rescale gives $\mathbb{E}[\hat{h}] = h$, so the sparsified delta is an unbiased approximation of the original in expectation. This formula converts singular-value magnitude into per-vector sparsity while keeping the target global sparsity $\alpha$; the allocation algorithm is written out for square matrices in the appendix.

What would settle it

Run Algorithm 1 on a real rectangular LLaMA layer, such as an MLP up-projection with $m \neq n$, at a target compression ratio of 32 and count the stored nonzeros including the deterministic-mask overhead; if the realized compression falls below the reported value or below DARE's realized ratio at the same accuracy, the central $2\times$ claim is refuted.

Watch

Extended reading notes

Core claim

The paper introduces ImPart and claims it is a state-of-the-art delta-sparsification method. Its central claim is that sparsifying delta parameters in SVD space with per-singular-vector sparsity ratios set by singular-value importance, rather than DARE's random dropping or LowRank's coarse truncation, preserves task-specific abilities at high sparsity. On GSM8K, MATH, HumanEval, MBPP, IFEval, and AlpacaEval across LLaMA2-13B, LLaMA2-7B, and LLaMA3-8B, ImPart outperforms DARE by 4.01 points on average and LowRank by 2.11, and at the same performance level reaches about $2\times$ the compression ratio. The same mechanism, combined with $\Delta$-CoMe, gives ImPart-QT the best delta-quantization scores, and used before Task Arithmetic or TIES merging it improves merged-model averages.

Load-bearing premise

The load-bearing assumption is that the sparsity-allocation scheme and its compression-ratio formula, which the paper writes out only for square matrices, extend to the rectangular MLP and attention layers of the LLaMA models tested without changing the reported ratios.

Editorial extensions

If this is right

  • At the same accuracy level, ImPart achieves roughly twice DARE's compression ratio on GSM8K and IFEval and about three times on HumanEval, so the same model quality can be stored in about half the space.
  • At a compression ratio of 16 (93.75% sparsity), the sparsified model retains 95.8% of the fine-tuned model's average performance, indicating that very sparse deltas remain practically usable.
  • ImPart-QT sets the best combined sparsification-plus-quantization numbers in the paper: 36.98 average versus 35.17 for Delta-CoMe, 36.12 for DARE-QT, and 32.71 for BitDelta at $CR_{qt}=32$.
  • Pre-sparsifying with ImPart before Task Arithmetic or TIES merging improves merged-model averages, so delta compression and model merging are compatible rather than competing goals.
  • Because the sparsity mask can be regenerated from the singular value's random seed, no mask storage is needed, so the reported compression ratio is not inflated by side information.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication not drawn in the paper: since the rescaling makes the reconstruction unbiased in expectation, the observed quality loss at high sparsity should be governed by the variance of the randomized reconstruction, so measuring that variance per layer could predict which layers need lower sparsity without a validation sweep.
  • The square-matrix presentation of Algorithm 1 means the $2\times$ claim should be re-derived for rectangular LLaMA layers; a direct check is to count stored nonzeros after applying Algorithm 1 to a real MLP or attention delta and compare with the reported compression ratio.
  • The authors note that validation-set hyperparameters can be misaligned with test tasks, so a natural follow-up is to test whether a single $\beta,C$ pair learned on math transfers to code and chat, which would remove the per-task tuning cost.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ImPart, an SVD-based delta-sparsification method for LLMs. It allocates a per-singular-vector sparsity ratio p_k that decreases with the singular value magnitude, pre-prunes the long tail of small singular components, and rescales surviving entries by 1/(1−p_k). The authors prove that the expectation of the reconstructed hidden state is preserved and present experiments on math, code, and chat tasks showing that ImPart outperforms DARE and a low-rank baseline at reported compression ratios, and that it composes with Delta-CoMe quantization and with task-arithmetic/TIES merging. The abstract claims a 2× higher compression ratio than baselines at the same performance level.

Significance. If the reported compression ratios are correct, ImPart would be a practical and simple improvement over DARE-style delta sparsification, with ablations and sensitivity analysis supporting the design. The paper is clearly written and includes public code. The main contribution is empirical; the expectation-preservation proof is a direct extension of DARE's argument. However, the paper's central quantitative claim (the CR values and the 2× advantage) depends on two currently unverified elements: the correctness of Algorithm 1's sparsity enforcement and a non-square generalization for LLaMA's rectangular layers. Both are fixable, but they need to be addressed before the results can be taken as stated.

major comments (3)
  1. [Appendix A.1, Algorithm 1, line 7] The while-loop condition checks (1/r)Σ_{k=1}^r p_k < α, but after line 1, α denotes the target average sparsity of the full U and V matrices. Since the tail components are pre-pruned to p_k = 1, the retained block must average (α−β)/(1−β), which is strictly smaller than α for β > 0. As written, the loop therefore continues past the intended sparsity and sets additional p_i to 1 until the retained-block average reaches α, overshooting the target. For α_orig = 31/32 (CR=32) and β = 0.6, the intended retained average is 0.9219, while the printed loop terminates only when it reaches 0.96875, so the actual U/V sparsity exceeds the intended value and the resulting compression ratio differs from the reported CR. The condition should be (1/r)Σ p_k < (α−β)/(1−β). This affects every CR in Tables 1, 5, 6, and 7, because Algorithm 1 maps a requested α to the per-vector sparsity ratios used throughout the paper.
  2. [Appendix A.1 and Section 3.2] The sparsity allocation and the compression-ratio derivation are presented only for square matrices ('For simplicity, we only present the case of square matrices' in A.1 and B.2), but all LLaMA layers used in the experiments are rectangular (e.g., the MLP projections are 4096×11008 or 5120×13824). For an m×n weight, the factor storage is r(m+n) for the r unpruned singular components, not 2n^2, and the relationship between per-matrix sparsity and the overall delta sparsity α changes with the aspect ratio m:n. The paper does not specify how sparsity ratios are allocated across the left and right singular vectors when m ≠ n, nor how CR is computed in that case. Because the abstract's '2× higher compression ratio' and the CR columns in Tables 1 and 5 depend on this quantity, the central quantitative claim is not anchored unless a correct rectangular-case accounting is provided and verified.
  3. [Abstract and Figure 1] The '2× higher compression ratio than baselines at the same performance level' claim is not backed by a direct iso-performance measurement. The tables report performance at fixed CR values, not the CR needed to reach a given performance level. For example, Table 5 shows that at CR=32 IMPART scores 60.20 on GSM8K while DARE scores 56.63; the CR at which DARE would reach 60.20 is not reported, so the factor of 2 is an interpretation of the plotted curves rather than a measured quantity. This headline claim should either be supported by explicit iso-performance CR extraction (e.g., from fine-grained CR sweeps with interpolation) or softened to a statement about performance at matched CR.
minor comments (5)
  1. [Equation (7)] The formula for p_k is ambiguous: it reads p_k = 1 − (σ_k/σ_1)^C · γ, which can be read as 1 − γ (σ_k/σ_1)^C, while Algorithm 1 line 5 uses p_i = (1 − (σ_i/σ_1)^C) γ. Please align the main-text formula with the algorithm.
  2. [Section 3.3, Equation (9)] The derivation has a notational slip: E[\hat W^{ft} X_j] should be E[\sum_j \hat W^{ft}_{ij} X_j], and the final sentence says the reconstructed embedding 'approximates the origin,' which should be 'approximates the original.'
  3. [Section 5.2] The validation set for mathematics is written 'SV AMP'; this appears to be SVAMP (a well-known math word problem benchmark).
  4. [Section 1 and Section 3.3] The theoretical result only establishes preservation of the expectation of the hidden state, not of the full output distribution or variance. The text in Section 1 that re-scaling 'ensures the performance is maintained' is stronger than what the proof supports; please soften this to 'preserves the expectation of the output.'
  5. [Tables 6 and 7] The method name is spelled inconsistently as IMPART-QT and IMPART-Qt; please use one convention throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the expectation-preservation proof is an algebraic identity, and the allocation and compression-ratio claims are empirical.

full rationale

The paper's only formal derivation is the expectation-preservation argument in Section 3.3. Equations (8)-(9) show that Bernoulli masking with rescaling coefficient 1/(1-p_k) makes E[\hat U_ik] = U_ik and E[\hat V_kj] = V_kj, so E[\widehat{\Delta W}] = \Delta W. This is a direct algebraic identity that is self-contained; it does not depend on the importance-aware allocation schedule in Eq. (7), which is a heuristic with hyperparameters (beta, C) selected on a validation set. The compression ratio is defined as CR = 1/(1-alpha), where alpha is the target sparsity input, so reporting CR values is a convention rather than a prediction derived from the method. The improvements over DARE and LowRank (Sections 5.5, 6.3, 7) are empirical comparisons against external baselines, not consequences of a fitted parameter being renamed as a result. Citations to prior work, including Delta-CoMe and SeTAR, are used as baselines, motivation, or suggestions for future work, not as load-bearing uniqueness/equivalence theorems. Concerns about the square-matrix-only pseudocode in Appendix A.1, the rectangular shapes of LLaMA layers, and the exact threshold in Algorithm 1 are implementation/correctness risks, not circularity, because the claimed CR tables are empirical reports and the sparsity target is an input rather than an output of the derivation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method rests on two tuned hyperparameters (beta and C), on the assumption that singular value magnitude is a valid importance proxy, on the assumption that expectation preservation implies preserved downstream performance, and on an unstated generalization to rectangular matrices. No new theoretical entities are introduced.

free parameters (2)
  • Pre-prune ratio beta = Searched over {0.6, 0.7, 0.8} on validation; e.g., 0.6 for math/code and chat in Table 4
    Determines the fraction of tail singular components removed entirely (Section 3.2, Eq. 7). Selected per task on validation sets (SVAMP, Mercury, FollowBench).
  • Regularization exponent C = Searched over {0.5, 1} on validation; 1 for math/code, 0.5 for chat in Table 4
    Shapes the relative sparsity among retained singular vectors. Selected per task on validation sets.
assumptions (4)
  • domain assumption Singular vectors with larger singular values encode more important task-specific information.
    Invoked in Section 3.1 and Eq. 7 to justify allocating lower sparsity to larger singular components. Based on prior observations (Ping et al., 2024; Sharma et al., 2024; Ryu et al., 2023), not proven for these tasks.
  • domain assumption Random dropping with 1/(1-p) rescaling preserves model output quality, not just expectation.
    Section 3.3 proves E[ΔŴ] = ΔW, but the paper asserts this maintains downstream task performance. The step from linear expectation to end-to-end performance is unproven and empirically supported only by the experiments.
  • ad hoc to paper The sparsity allocation and compression-ratio computation extend to non-square weight matrices.
    Algorithm 1 is stated for square matrices only. The experiments use rectangular layers (e.g., LLaMA MLP), so the method implicitly assumes a valid generalization without specifying it.
  • domain assumption Validation benchmarks are predictive of test performance for hyperparameter selection.
    Section 5.2 tunes beta and C on SVAMP/Mercury/FollowBench; Section 6.2 admits this sometimes selects suboptimal configurations (e.g., IFEval), so the assumption is imperfect but the paper still reports gains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs." pith.science (2026). https://pith.science/paper/5MSJ6EEU

@misc{pith2026250413237,
  author       = {Pith},
  title        = {Pith review of: ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5MSJ6EEU}},
  note         = {Machine review of arXiv:2504.13237}
}
abstract

With the proliferation of task-specific large language models, delta compression has emerged as a method to mitigate the resource challenges of deploying numerous such models by effectively compressing the delta model parameters. Previous delta-sparsification methods either remove parameters randomly or truncate singular vectors directly after singular value decomposition (SVD). However, these methods either disregard parameter importance entirely or evaluate it with too coarse a granularity. In this work, we introduce ImPart, a novel importance-aware delta sparsification approach. Leveraging SVD, it dynamically adjusts sparsity ratios of different singular vectors based on their importance, effectively retaining crucial task-specific knowledge even at high sparsity ratios. Experiments show that ImPart achieves state-of-the-art delta sparsification performance, demonstrating $2\times$ higher compression ratio than baselines at the same performance level. When integrated with existing methods, ImPart sets a new state-of-the-art on delta quantization and model merging.

Figures

Figures reproduced from arXiv: 2504.13237 by the authors.

Figure 1
Figure 1. Comparative evaluation of IMPART against state-of-the-art sparsification methods across mathematical reasoning, code generation, and chat tasks. IMPART consistently outperforms baselines across various tasks while maintaining high sparsity ratios (more detailed discussions are in Section 6.3). across three diverse tasks with various backbones demonstrate the effectiveness of our method. As shown in [PITH_FULL_IMAGE… view at source ↗
Figure 2
Figure 2. Overview of IMPART. (a) Delta parameters computation by subtracting the base model from the fine-tuned model. (b) Comparison of delta parameters sparsification methods: DARE randomly drops delta parameters, LowRank sparsifies with low-rank approximation, and IMPART adaptively sparsifies singular vectors. (c) Further apply mixed-precision quantization on sparse singular vectors to achieve higher compression ratios. (… view at source ↗
Figure 3
Figure 3. Importance-aware delta-sparsification adap [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparative evaluation of IMPART against state-of-the-art quantization methods across mathematical reasoning, code generation, and chat tasks (more detailed discussions are in Section 7.1). the 1-bit element-wise sign of each parameter and µ is a trainable scaling fact…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 14 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, et al. 2024. https://arxiv.org/abs/2412.08905 Phi-4 technical report . Preprint, arXiv:2412.08905

    arXiv 2024

  2. [2]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. https://arxiv.org/abs/2108.07732 Program synthesis with large language models . Preprint, arXiv:2108.07732

    arXiv 2021

  3. [3]

    Mckinstry, Steven K

    Deepika Bablani, Jeffrey L. Mckinstry, Steven K. Esser, Rathinakumar Appuswamy, and Dharmendra S. Modha. 2024. https://arxiv.org/abs/2301.13330 Efficient and effective methods for mixed precision neural network quantization for faster, energy-efficient inference . Preprint, arXiv:2301.13330

    arXiv 2024

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al. 2021. https://arxiv.org/abs/2107.03374 Evaluating large language models trained on code . Preprint, arXiv:2107.03374

    arXiv 2021

  5. [5]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    arXiv 2021

  6. [6]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, et al. 2025. https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning . Preprint, arXiv:2501.12948

    arXiv 2025

  7. [7]

    Zhang, and Han Bao et al

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, and Han Bao et al. 2024. https://arxiv.org/abs/2412.19437 Deepseek-...

    arXiv 2024

  8. [8]

    Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. 2023. https://arxiv.org/abs/2306.03078 Spqr: A sparse-quantized representation for near-lossless llm weight compression . Preprint, arXiv:2306.03078

    arXiv 2023

Show all 48 references
  1. [9]

    Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, and Min Zhang. 2024 a . https://proceedings.neurips.cc/paper_files/paper/2024/file/99fc8bc48b917c301a80cb74d91c0c06-Paper-Conference.pdf Parameter competit...

    2024

  2. [10]

    Mingzhe Du, Anh Tuan Luu, Bin Ji, Qian Liu, and See-Kiong Ng. 2024 b . https://openreview.net/forum?id=vyraA7xt4c Mercury: A code efficiency benchmark for code large language models . In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track

    2024

  3. [11]

    Hashimoto

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. https://arxiv.org/abs/2404.04475 Length-controlled alpacaeval: A simple way to debias automatic evaluators . Preprint, arXiv:2404.04475

    2024 arXiv

  4. [12]

    Razvan-Gabriel Dumitru, Paul-Ioan Clotan, Vikas Yadav, Darius Peteleaza, and Mihai Surdeanu. 2024. https://arxiv.org/abs/2411.03513 Change is the only constant: Dynamic llm slicing based on layer redundancy . Preprint, arXiv:2411.03513

    2024 arXiv

  5. [13]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. https://openreview.net/forum?id=tcbBPnfwxS OPTQ : Accurate quantization for generative pre-trained transformers . In The Eleventh International Conference on Learning Representations

    2023

  6. [14]

    Shangqian Gao, Ting Hua, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. 2024. https://doi.org/10.18653/v1/2024.naacl-long.13 Adaptive rank selections for low-rank approximation of language models . In NAACL-HLT, pages 227--241

    2024 doi

  7. [15]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  8. [16]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=7Bywt2mQsCe Measuring mathematical problem solving with the MATH dataset . In Thirty-fifth Conference on Neural Informatio...

    2021

  9. [17]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. https://openreview.net/forum?id=6t0Kwf8-jrj Editing models with task arithmetic . In The Eleventh International Conference on Learning Representations

    2023

  10. [18]

    Berivan Isik, Hermann Kumbong, Wanyi Ning, Xiaozhe Yao, Sanmi Koyejo, and Ce Zhang. 2023. https://openreview.net/forum?id=hO0c2tG2xL GPT -zip: Deep compression of finetuned large language models . In Workshop on Efficient Systems for Foundation Models @ ICML2023

    2023

  11. [19]

    Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, et al. 2023. https://arxiv.org/abs/2310.06825 Mistral 7b . Preprint, arXiv:2310.06825

    2023 arXiv

  12. [20]

    Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024. https://doi.org/10.18653/v1/2024.acl-long.257 F ollow B ench: A multi-level fine-grained constraints following benchmark for large language models ....

    2024 doi

  13. [21]

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2023. https://openreview.net/forum?id=FCnohuR6AnM Dataless knowledge fusion by merging weights of language models . In The Eleventh International Conference on Learning Representations

    2023

  14. [22]

    Kimi Team et al. 2025. https://arxiv.org/abs/2501.12599 Kimi k1.5: Scaling reinforcement learning with llms . Preprint, arXiv:2501.12599

    2025 arXiv

  15. [23]

    Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. 2021. https://openreview.net/forum?id=H6ATjJ0TKdf Layer-adaptive sparsity for the magnitude-based pruning . In International Conference on Learning Representations

    2021

  16. [24]

    Guiying Li, Chao Qian, Chunhui Jiang, Xiaofen Lu, and Ke Tang. 2018. https://doi.org/10.24963/ijcai.2018/330 Optimization based layer-wise magnitude-based pruning for dnn compression . In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligen...

    2018 doi

  17. [25]

    Yixia Li, Boya Xiong, Guanhua Chen, and Yun Chen. 2024. https://openreview.net/forum?id=65UoJ0z7Kp Se TAR : Out-of-distribution detection with selective low-rank approximation . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  18. [26]

    Zhiteng Li, Mingyuan Xia, Jingyuan Zhang, Zheng Hui, Linghe Kong, Yulun Zhang, and Xiaokang Yang. 2025. https://arxiv.org/abs/2502.01403 Adasvd: Adaptive singular value decomposition for large language models . Preprint, arXiv:2502.01403

    2025

  19. [27]

    Lee, Song Han, Tri Dao, and Tianle Cai

    James Liu, Guangxuan Xiao, Kai Li, Jason D. Lee, Song Han, Tri Dao, and Tianle Cai. 2024. https://arxiv.org/abs/2402.10193 Bitdelta: Your fine-tune may only be worth one bit . Preprint, arXiv:2402.10193

    2024 arXiv

  20. [28]

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. 2025. https://arxiv.org/abs/2308.09583 Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-...

    2025 arXiv

  21. [29]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. https://arxiv.org/abs/2306.08568 Wizardcoder: Empowering code large language models with evol-instruct . Preprint, arXiv:2306.08568

    2023 arXiv

  22. [30]

    Michael Matena and Colin Raffel. 2022. https://arxiv.org/abs/2111.09832 Merging models with fisher-weighted averaging . Preprint, arXiv:2111.09832

    2022 arXiv

  23. [31]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. https://doi.org/10.18653/v1/2021.naacl-main.168 Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling...

    2021 doi

  24. [32]

    Bowen Ping, Shuo Wang, Hanqing Wang, Xu Han, Yuzhuang Xu, Yukun Yan, Yun Chen, Baobao Chang, Zhiyuan Liu, and Maosong Sun. 2024. https://openreview.net/forum?id=cr5EQRJlRn Delta-come: Training-free delta-compression with mixed-precision for large language models . In The Thirt...

    2024

  25. [33]

    Simo Ryu, Seunghyun Seo, and Jaejun Yoo. 2023. https://arxiv.org/abs/2305.18425 Efficient storage of fine-tuned models via low-rank approximation of weight residuals . Preprint, arXiv:2305.18425

    2023 arXiv

  26. [34]

    Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea Goldsmith, and Mert Pilanci. 2024. https://openreview.net/forum?id=lkx3OpcqSZ Compressing large language models using low rank and low precision decomposition . In The Thirty-eighth Annual Conference on Neural Information Pr...

    2024

  27. [35]

    Ash, and Dipendra Misra

    Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. 2024. https://openreview.net/forum?id=ozX92bu8VA The truth is in there: Improving reasoning in language models with layer-selective rank reduction . In The Twelfth International Conference on Learning Representations

    2024

  28. [36]

    Xuan Shen, Peiyan Dong, Lei Lu, Zhenglun Kong, Zhengang Li, Ming Lin, Chao Wu, and Yanzhi Wang. 2023. https://arxiv.org/abs/2312.05693 Agile-quant: Activation-guided quantization for faster inference of llms on the edge . Preprint, arXiv:2312.05693

    2023 arXiv

  29. [37]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, et al. 2023. https://arxiv.org/abs/2307.09288 Llama 2: Open foundation and fine-tuned chat models . Preprint, arXiv:2307.09288

    2023 arXiv

  30. [38]

    Boyao Wang, Rui Pan, Shizhe Diao, Xingyuan Pan, Jipeng Zhang, Renjie Pi, and Tong Zhang. 2025 a . https://arxiv.org/abs/2502.03460 Adapt-pruner: Adaptive structural pruning for efficient small language model training . Preprint, arXiv:2502.03460

    2025

  31. [39]

    Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. 2025 b . https://openreview.net/forum?id=LNYIUouhdt SVD - LLM : Truncation-aware singular value decomposition for large language model compression . In The Thirteenth International Conference on Learning Representations

    2025

  32. [40]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. https://arxiv.org/abs/2312.02120 Magicoder: Empowering code generation with oss-instruct . Preprint, arXiv:2312.02120

    2024 arXiv

  33. [41]

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. 2022. https://proceedings.mlr.press/v162/wortsman22a.html Model soups: averaging weights ...

    2022

  34. [42]

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. 2023. https://openreview.net/forum?id=xtaX3WyCj1 TIES -merging: Resolving interference when merging models . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  35. [43]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, et al. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  36. [44]

    Xiaozhe Yao, Qinghao Hu, and Ana Klimovic. 2024. https://arxiv.org/abs/2312.05215 Deltazip: Efficient serving of multiple full-model-tuned llms . Preprint, arXiv:2312.05215

    2024 arXiv

  37. [45]

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In International Conference on Machine Learning. PMLR

    2024

  38. [46]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. https://arxiv.org/abs/2311.07911 Instruction-following evaluation for large language models . Preprint, arXiv:2311.07911

    2023 arXiv

  39. [47]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.