Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

DLP: Dynamic Layerwise Pruning in Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DLP claims that median-based, activation-aware layer importance, converted into non-uniform sparsity rates, preserves LLM quality at high pruning levels where uniform and outlier-threshold allocation fail.

desk verdict DLP is a useful but incremental layerwise-pruning method whose headline gains over OWL likely owe something to tuning α on the evaluation set while OWL's M stays fixed. read the letter →

arxiv 2505.23807 v3 pith:6GRSYZZE submitted 2025-05-27 cs.CL cs.AI

Yuli Chen , Bo Cheng , Jiale Han , Yingying Zhang , Yingting Li , Shuhao Zhang This is my paper · ORCID
classification cs.CLcs.AI
keywords largelanguagemodelslayerwisepruningnon-uniformsparsityactivation-awareimportancemedian-basedhighLLMcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that large language models should be pruned layer by layer according to a measured relative importance, not with one uniform sparsity rate. It claims that taking the median of activation-aware weight scores in each layer gives a reliable unimportance signal, and that inverting this into per-layer pruning rates keeps perplexity and accuracy far closer to the dense model at 70-90% sparsity. The reported payoff is concrete: at 70% sparsity on LLaMA2-7B, perplexity drops by 7.79 and average zero-shot accuracy rises by 2.7% compared with state-of-the-art layerwise baselines, and pruned models run 2.8-3.7x faster on CPU. This matters because high-sparsity pruning is where uniform methods collapse, and a parameter-free way to allocate sparsity would make LLM compression cheaper and more predictable.

What carries the argument

The central object is the Relative Importance Distribution (RID), computed from the median of activation-aware weight scores across each Transformer layer. The median plays the load-bearing role: earlier median-based pruning work treats elements near a layer's center as replaceable, and the paper extends that premise from filters to LLM layers, using a higher median to mean more redundancy and hence a higher pruning rate. An $\alpha$-scaling step then compresses the importance range so no single layer is pruned too aggressively, mapping each layer's sparsity into $[R-\alpha, R+\alpha]$ around the global rate $R$.

What would settle it

Prune each layer of LLaMA2-7B individually to the same sparsity rate, measure the perplexity jump each layer causes, and compare that ranking with the sparsity DLP assigns; if the layers DLP prunes most aggressively are not the layers whose isolated removal hurts least, the median redundancy premise is falsified.

Watch

Extended reading notes

Core claim

DLP's central claim is that layer importance can be computed adaptively from the same weight-activation signal used by Wanda. For each layer $l$, it aggregates the score $A^l_{ij} = |W^l_{ij}| \cdot \|X^l_j\|_2$ with the median, forming an unimportance score $S_l = \operatorname{median}(A^l_{ij})$. The relative importance $I_l = 1 - S_l / \sum_i S_i$ is then scaled and inverted so that more important layers receive lower sparsity, with a hyperparameter $\alpha$ controlling how far individual layers deviate from the global sparsity rate. The paper reports that this non-uniform allocation consistently beats both uniform pruning and the outlier-proportion allocation of OWL, with the largest gains at high sparsity.

Load-bearing premise

The load-bearing premise is that the median of a layer's activation-aware weight scores measures how redundant that layer is, so layers with higher medians can safely receive heavier pruning.

Editorial extensions

If this is right

  • At 70-90% unstructured sparsity, the paper reports 2.8-3.7x end-to-end speedups on CPU decoding for LLaMA2-7B-chat with DLP-assigned rates, compared with the dense model, while preserving far more quality than uniform pruning.
  • Because DLP only changes how sparsity is distributed, it can be layered on top of Wanda, SparseGPT, and magnitude pruning, and the paper shows the same RID improves structured pruning, N:M sparsity, SVD, and quantization.
  • The paper shows that a short LoRA fine-tuning stage after pruning substantially recovers quality, with WikiText perplexity falling from 17.76 to 12.15 for LLaMA1-7B at 70% sparsity.
  • At sparsity rates below 40%, DLP's gains are small; the method's value concentrates in the high-sparsity territory where uniform and outlier-based allocation degrade sharply.
  • Deep layers receive higher sparsity than early layers under DLP, consistent with the paper's observation that later layers are more redundant for these models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The median ranking could be evaluated as a cheap layer-sensitivity estimator in its own right, for example by comparing DLP's per-layer rates with rates chosen by direct layer-removal or reconstruction-error probes on the same models.
  • Because $\alpha$ is tuned per sparsity level on validation data, a testable extension is whether a fixed or predicted $\alpha$ works across sparsity levels; if not, the method's parameter-free claim rests mainly on the median ranking itself.
  • Since RID is computed from activations, it may transfer to other transformer families or modalities where outlier channels dominate, so one could test whether median-based allocation helps vision or multimodal transformers at high pruning rates.
  • The large perplexity gains reported for mixed 1:4 and 2:8 sparsity suggest hardware-friendly N:M schedules could be selected dynamically from RID, but real kernel benchmarks are needed to confirm the speedups translate beyond perplexity improvements.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Dynamic Layerwise Pruning (DLP), a method for assigning non-uniform sparsity to the layers of large language models. DLP computes a per-layer unimportance score as the median of Wanda-style weight-activation products (Eq. 5), converts it into a relative importance via a linear inversion (Eq. 6), and then scales the importance range with a hyperparameter alpha so that each layer's sparsity lies in [R-alpha, R+alpha] (Algorithm 1, lines 9-13). The authors evaluate DLP combined with Magnitude, SparseGPT, and Wanda on LLaMA1, LLaMA2, LLaMA3, Vicuna, Mistral, and Qwen models at sparsity levels from 10% to 90%, using WikiText, PTB, and C4 perplexity and seven zero-shot tasks. They report consistent improvements over uniform pruning and over the OWL baseline, especially at 70% sparsity (e.g., a 7.79 perplexity reduction and a 2.7% accuracy gain on LLaMA2-7B), as well as CPU speedups and integrations with quantization, N:M sparsity, SVD, and PEFT.

Significance. If the reported results are reliable, DLP would be a practical and broadly applicable layerwise sparsity allocator for high-sparsity LLM pruning. The paper has several strengths: it covers a wide range of models, datasets, and sparsity levels; it shows compatibility with multiple compression techniques; it reports end-to-end inference speedups; and it releases code. However, the central quantitative claims are weakened by an asymmetric comparison with OWL and by the fact that the method's key hyperparameter is tuned on the same evaluation metric used to report improvements. These issues need to be resolved before the paper's conclusions can be accepted as stated.

major comments (3)
  1. [Section G, Table 19] The hyperparameter alpha is selected by iterative experiments on the WikiText validation set for each sparsity level, and the main results in Tables 3 and 10 report WikiText validation perplexity. This makes the reported improvements in-sample selection results rather than out-of-sample evaluations. By contrast, the OWL baseline uses a fixed M=7 (Section 3.3) with no per-model tuning, even though Figure 2 shows that the optimal M varies across model families and scales. The comparison is therefore not symmetric, and the headline gains (e.g., the 7.79 perplexity reduction on LLaMA2-7B) may reflect unequal hyperparameter optimization rather than a genuinely better importance signal. The paper should either hold out the evaluation set when tuning alpha or compare DLP against OWL with M also tuned per model on the same validation set.
  2. [Section 3.3, Table 1] The choice of F as the median in Eq. (5) is justified solely by Table 1, which reports results for one model (LLaMA1-7B) at one sparsity level (70%) on one dataset (WikiText). Because the same dataset is used both to select F and to report the final comparisons in Table 3, this is a model-selection step rather than a validation. The paper provides no evidence that the median remains the best aggregator across different models, sparsity levels, or datasets; given that this choice is load-bearing for the entire method, it needs support from experiments over multiple configurations or a principled argument for why the median should generalize.
  3. [Section 4.2, Tables 3-4] No measure of variability is reported: there are no standard deviations, confidence intervals, or multiple seeds. At sparsity levels where DLP's advantage is small (e.g., Table 10, 50% sparsity with SparseGPT: 7.15 vs 7.19), the differences are comparable to typical run-to-run variation in perplexity evaluation. Without repeated runs or error bars, the claim of consistent improvement over OWL and uniform pruning is not statistically supported, particularly in the moderate-sparsity regime where the absolute gaps are tiny.
minor comments (6)
  1. [Section 3.3, Eq. (5)] The notation in Eq. (5) is ambiguous: F is applied to A_ij, but the double sum suggests F is applied after aggregation. Please clarify that F (the median) is computed over all entries in the layer's score matrix, and then the sum is taken (or that the median is of the set of all A_ij values).
  2. [Algorithm 1] The variable `d` is used both as a stored list and in the expression `d_j`; lines 9-10 refer to `I_i` without specifying the index range, and `m` is defined as the mean of `d` after the loop. Please make the indexing and the storage of scaled scores explicit.
  3. [Section 1, contributions] The claim that the method 'does not rely on empirical values or model type' is contradicted by the hyperparameter alpha and the grid search described in Section G; alpha is an empirical value that is tuned per sparsity level. Please rephrase this contribution to acknowledge the role of alpha.
  4. [Section G, Table 19] The selection protocol for alpha is described only as 'iterative experiments.' Please state explicitly that alpha was chosen by minimizing WikiText validation perplexity, so that readers can assess the degree of selection and reproduce the protocol.
  5. [Table 1] The numerical formatting in Table 1 is inconsistent (e.g., '3.7e3 3.7e3' versus '3.4e3' in the Magnitude row), and the Sum and Mean columns appear to contain identical values for several rows. Please verify these entries and use a consistent number of significant digits.
  6. [Appendix B] The sentence 'DLP reduces perplexity by 65.92 compared to uniform layerwise pruning' refers to numbers that appear only in Table 10; please make the referenced values explicit in the text for readability.

Circularity Check

3 steps flagged · score 4.0 of 10

DLP's headline perplexity gains are partly post-selection: the median unimportance metric and the α scaling are chosen on the same WikiText perplexity used to report improvements, while the RID non-uniformity evidence is a tautological property of its own definition.

  1. fitted input called prediction [Section 3.3, Empirical Study I; Table 1]
    "As presented in Table 1, the median method performs better than other methods. ... Therefore, we choose F(·) as the median."

    The aggregation function F in Eq. (5) is not derived from first principles; it is selected by ranking six candidates (Sum, Mean, Median, Max, Var, SD) on WikiText validation perplexity of LLaMA1-7B at 70% unstructured sparsity, as stated in the Table 1 caption. The paper then reports DLP's success on the same WikiText perplexity metric, e.g., 'Wanda reduces it by 7.79 on LLaMA2-7B.' For the configuration used in the selection, the reported perplexity is the selected outcome of the Table 1 search rather than an independent prediction; the choice of median is a fit to the evaluation metric used to validate the method.

  2. fitted input called prediction [Section G, Hyperparameter Setting; Tables 19 and 3]
    "To determine the matching relationship between different sparsity levels and relative importance, we conduct iterative experiments with α∈[0.02,0.04,0.06,0.08,0.1,0.12,0.15,0.2]. We provide the hyperparameter setting for different sparsity levels to facilitate the reproduction of our method's results, as shown in Table 19."

    α is the deflation scale that controls the range of layerwise sparsity (Algorithm 1, lines 9-13). Section G states that α is selected by iterative grid experiments, and Table 19 lists the selected values per sparsity level (α=0.15 at 70%). The paper's main perplexity tables (Tables 3, 9, 10) are all WikiText validation perplexities, the same metric used to select α. The headline claim that 'at 70% sparsity, DLP reduces the perplexity of LLaMA2-7B by 7.79' is therefore a post-selection number obtained by tuning a hyperparameter on the evaluation set, not an out-of-sample prediction.

1 more flagged steps
  1. self definitional [Section 3.3, Empirical Study II; Eq. (6); Figure 3]
    "If RID is highly balanced, it indicates that our method is not suitable for evaluating inter-layer importance. As shown in the bar chart in the background of Figure 3, the results show that not all layers contribute equally to the model's performance."

    RID is defined by Eqs. (5)-(6) as I_l = 1 - S_l / Σ S_l, where S_l is the median of Wanda scores in layer l. Non-uniformity of RID is guaranteed whenever layer medians differ, because I_l is just a normalized inverse transform of those medians. Using the non-uniformity of this constructed quantity as evidence that 'not all layers contribute equally to the model's performance' validates the proxy by its own definition; no independent performance measurement is involved.

full rationale

DLP is not circular in the strong sense: its layerwise sparsity allocation is computed from the median of Wanda scores and then inverted and rescaled, while the reported perplexity is an actual evaluation of the pruned model. However, two choices that carry the method's claimed advantage are selected on the very WikiText perplexity metric used to report the headline gains: the choice of F(·)=median comes from a six-way comparison on WikiText validation perplexity at 70% sparsity on LLaMA1-7B, and α is tuned by iterative experiments on the same evaluation-style perplexity. Consequently, at least part of the 'reduces perplexity by 7.79' headline is a selected outcome rather than a prediction. In addition, Empirical Study II treats the non-uniformity of RID as evidence that layers contribute unequally, but that non-uniformity follows by construction from the definition of RID. The external benchmark results, zero-shot accuracy, other datasets (PTB, C4), and other model families provide independent content and mitigate the circularity; the fixed M=7 comparison with OWL is a fairness concern rather than a circularity. Overall, a moderate partial-circularity score of 4 is appropriate.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

DLP introduces no invented entities. The only fitted numeric parameter is alpha, which is selected for each sparsity level from a grid using the same validation data on which performance is then reported. The median unimportance metric and the linear importance-to-sparsity mapping are modeling assumptions inherited from empirical observation, not derived from first principles.

free parameters (1)
  • alpha (deflation scale) = 0.06, 0.02, 0.04, 0.02, 0.04, 0.1, 0.15, 0.12 for 10%-80% sparsity
    Controls the range [R-alpha, R+alpha] for per-layer sparsity. Chosen by grid search on the evaluation perplexity, so the reported results depend on fitted values.
assumptions (3)
  • domain assumption Median of Wanda scores measures layer redundancy
    Eq. (5) and Table 1: median is selected after comparing five summary statistics on LLaMA1-7B at 70% sparsity. No derivation is given for why the median captures redundancy.
  • domain assumption Wanda score is a valid weight importance metric
    Eq. (3): DLP inherits Wanda's |W| times activation norm as base importance. If this criterion fails, the layer-level median inherits the failure.
  • ad hoc to paper Linear inversion and alpha-bounded scaling map importance to sparsity
    Eq. (6) and Algorithm 1 lines 9-13: the inversion I=1-S/sum(S), the min-max rescaling to [0,2alpha], and R=p+m-d are heuristic design choices without an optimality argument.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DLP: Dynamic Layerwise Pruning in Large Language Models." pith.science (2026). https://pith.science/paper/6GRSYZZE

@misc{pith2026250523807,
  author       = {Pith},
  title        = {Pith review of: DLP: Dynamic Layerwise Pruning in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6GRSYZZE}},
  note         = {Machine review of arXiv:2505.23807}
}
read the original abstract

Pruning has recently been widely adopted to reduce the parameter scale and improve the inference efficiency of Large Language Models (LLMs). Mainstream pruning techniques often rely on uniform layerwise pruning strategies, which can lead to severe performance degradation at high sparsity levels. Recognizing the varying contributions of different layers in LLMs, recent studies have shifted their focus toward non-uniform layerwise pruning. However, these approaches often rely on pre-defined values, which can result in suboptimal performance. To overcome these limitations, we propose a novel method called Dynamic Layerwise Pruning (DLP). This approach adaptively determines the relative importance of each layer by integrating model weights with input activation information, assigning pruning rates accordingly. Experimental results show that DLP effectively preserves model performance at high sparsity levels across multiple LLMs. Specifically, at 70% sparsity, DLP reduces the perplexity of LLaMA2-7B by 7.79 and improves the average accuracy by 2.7% compared to state-of-the-art methods. Moreover, DLP is compatible with various existing LLM compression techniques and can be seamlessly integrated into Parameter-Efficient Fine-Tuning (PEFT). We release the code at https://github.com/ironartisan/DLP to facilitate future research.

Figures

Figures reproduced from arXiv: 2505.23807 by the authors.

Figure 1
Figure 1. Illustration of Uniform Layerwise Pruning and Dynamic Layerwise Pruning (DLP): Blue squares represent unpruned weights, while white squares denote pruned weights. In uniform layerwise pruning, the same sparsity ratio is applied to every layer. In contrast, DLP calculates the unimportance of each Transformer block to compare the relative importance of layers, assigning different sparsity ratios based on the principle… view at source ↗
Figure 2
Figure 2. WikiText validation perplexity of LLaMA1-7B, LLaMA1-13B and Vicuna-7B pruned by various M at 70% sparsity using OWL. 3.3. Empirical Study Relative Importance Distribution (RID). We use RID as the basis for assigning layer sparsity rates. RID takes into ac￾count both intra-layer and inter-layer element importance within LLMs. For intra-layer analysis, RID compares the ab￾solute unimportance of Transformer blocks with… view at source ↗
Figure 3
Figure 3. Comparison of layerwise sparsity distributions between Ours (red) and OWL (orange). The bar chart in the background represents the Relative Importance Distribution (RID). In each subplot, the horizontal axis represents the layer index, the left vertical axis corresponds to the RID, and the right vertical axis corresponds to the layerwise sparsity ratio [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of different methods at high sparsity, Using SparseGPT and Wanda [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Comparison of per-output and per-layer perplexity at different sparsity rates. Due to the significant difference in values between high and low sparsity rates, we use the logarithmic results for comparison. pruned with DLP consistently outperforms those with uniform sp…
Figure 6
Figure 6. Figure 6: Perplexity of LLaMA1-7B on different validation datasets under varying quantization bits [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Visual-token attention filtering improves structured pruning of vision-language models, keeping 94% of average benchmark accuracy after removing 20% of parameters.

  2. SkipOPU: An FPGA-based Overlay Processor for Large Language Models with Dynamically Allocated Computation

    cs.AR 2026-03 conditional novelty 6.0 of 10

    An FPGA accelerator for SkipGPT-style pruned LLMs that fuses router/normalization/attention computation to hide nonlinear latency, packs two FP16 products in each DSP, and serves reused KV entries from on-chip URAM to...

  3. Are the High-weight Neurons the Important Ones in Image Classification Neural Networks?

    cs.AI 2026-07 reject novelty 3.0 of 10

    Weight magnitude is a weak and nonlinear proxy for per-weight importance in CNNs, but the paper's quantitative claims are undermined by a mislabeled metric and an unconventional definition of 'neuron'.

Reference graph

Works this paper leans on

21 extracted references · 4 canonical work pages · cited by 3 Pith papers

  1. [2]

    URL https://openreview.net/forum? id=4IWCHWlb6K. Boratko, M., Padigela, H., Mikkilineni, D., Yuvraj, P., Das, R., McCallum, A., Chang, M., Fokoue-Nkoutche, A., Ka- panipathi, P., Mattei, N., Musa, R., Talamadupula, K., and Witbrock, M. A systematic classification of knowl- edge, reasoning, and context within the ARC dataset. In Choi, E., Seo, M., Chen, D....

    2018

  2. [3]

    Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y ., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y ., Gonzalez, J

    URL https://openreview.net/forum? id=9vKRhnflAs. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y ., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y ., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. Clark, C., Lee, K., Chang, M., Kwiatkowski, T., Collins, M., and Toutanova, K...

    2023

  3. [4]

    URL https://openreview.net/forum? id=1tRLxQzdep. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozi `ere, B., Biron, B., Tang, B....

    arXiv

  4. [5]

    URL https: //doi.org/10.48550/arXiv.2407.21783

    doi: 10.48550/ARXIV .2407.21783. URL https: //doi.org/10.48550/arXiv.2407.21783. Fan, S., Jiang, X., Li, X., Meng, X., Han, P., Shang, S., Sun, A., Wang, Y ., and Wang, Z. Not all layers of llms are necessary during inference.CoRR, abs/2403.02181,

    arXiv doi:10.48550/arxiv.2407.21783

  5. [6]

    URL https: //doi.org/10.48550/arXiv.2403.02181

    doi: 10.48550/ARXIV .2403.02181. URL https: //doi.org/10.48550/arXiv.2403.02181. Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In7th Inter- national Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net, 2019. URL https://openreview.net/ forum?id=rJl-...

    arXiv doi:10.48550/arxiv.2403.02181

  6. [9]

    Huber, K

    URL https://openreview.net/forum? id=nZeVKeeFYf9. Huber, K. T., Moulton, V ., Lockhart, P., and Dress, A. Pruned median networks: a technique for reducing the complexity of median networks.Molecular phylogenetics and evolution, 19(2):302–310, 2001. Jaiswal, A., Liu, S., Chen, T., and Wang, Z. The emergence of essential sparsity in large pre-trained models...

    arXiv 2001

  7. [10]

    URL https: //doi.org/10.48550/arXiv.2310.06825

    doi: 10.48550/ARXIV .2310.06825. URL https: //doi.org/10.48550/arXiv.2310.06825. Kurtic, E., Kuznedelev, D., Frantar, E., Goin, M., and Alistarh, D. Sparse fine-tuning for inference accelera- tion of large language models.CoRR, abs/2310.06927,

    arXiv doi:10.48550/arxiv.2310.06825

  8. [11]

    Sparse Fine-tuning for Inference Acceleration of Large Language Models

    doi: 10.48550/ARXIV .2310.06927. URL https: //doi.org/10.48550/arXiv.2310.06927. Lee, C., Jin, J., Kim, T., Kim, H., and Park, E. OWQ: lessons learned from activation outliers for weight quantization in large language models.CoRR, abs/2306.02272, 2023. doi: 10.48550/ARXIV .2306.02272. URL https:// doi.org/10.48550/arXiv.2306.02272. Lee, J., Park, S., Mo, ...

    work page Pith review arXiv doi:10.48550/arxiv.2310.06927

Show all 21 references
  1. [12]

    Ling, Z., Li, Z., Romero, P., Han, L., and Nenadic, G

    mlsys.org, 2024. Ling, Z., Li, Z., Romero, P., Han, L., and Nenadic, G. Bee- manc at the PLABA track of TAC-2024: roberta for task 1 - llama3.1 and gpt-4o for task 2.CoRR, abs/2411.07381,

    2024 arXiv

  2. [13]

    URL https: //doi.org/10.48550/arXiv.2411.07381

    doi: 10.48550/ARXIV .2411.07381. URL https: //doi.org/10.48550/arXiv.2411.07381. Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. Few-shot parameter-efficient fine- tuning is better and cheaper than in-context learning. In 11 DLP: Dynamic Layerwi...

    doi arXiv

  3. [14]

    URL https: //doi.org/10.48550/arXiv.2303.08774

    doi: 10.48550/ARXIV .2303.08774. URL https: //doi.org/10.48550/arXiv.2303.08774. Pan, R., Liu, X., Diao, S., Pi, R., Zhang, J., Han, C., and Zhang, T. LISA: layerwise importance sampling for memory-efficient large language model fine-tuning. CoRR, abs/2403.17919, 2024. doi: 10...

    doi arXiv

  4. [17]

    Yin, L., Wu, Y ., Zhang, Z., Hsieh, C., Wang, Y ., Jia, Y ., Li, G., Jaiswal, A

    URL https://proceedings.mlr.press/ v202/xiao23c.html. Yin, L., Wu, Y ., Zhang, Z., Hsieh, C., Wang, Y ., Jia, Y ., Li, G., Jaiswal, A. K., Pechenizkiy, M., Liang, Y ., Bendersky, M., Wang, Z., and Liu, S. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pru...

    2024

  5. [19]

    URL https: //doi.org/10.1109/TCYB.2021.3124284

    doi: 10.1109/TCYB.2021.3124284. URL https: //doi.org/10.1109/TCYB.2021.3124284. Zhang, Y ., Bai, H., Lin, H., Zhao, J., Hou, L., and Can- nistraci, C. V . Plug-and-play: An efficient post-training pruning method for large language models. InThe Twelfth International Conference...

    2021

  6. [31]

    This is likely because such an approach creates significant sparsity discrepancies between blocks, potentially disrupting inter-layer information flow

    It is noteworthy that comparing importance at the level of each Transformer block results in varying sparsity levels across blocks, leading to suboptimal performance. This is likely because such an approach creates significant sparsity discrepancies between blocks, potentially...

    2023

  7. [2018]

    13 DLP: Dynamic Layerwise Pruning in Large Language Models A

    URL https://openreview.net/forum? id=Sy1iIDkPM. 13 DLP: Dynamic Layerwise Pruning in Large Language Models A. Comparison among Various Layerwise Sparsity Methods In Table 9, we further compare the performance of DLP with other layerwise sparsity methods on LLaMA1-7B. The detai...

    2019

  8. [2019]

    URL https:// doi.org/10.18653/v1/p19-1472

    doi: 10.18653/V1/P19-1472. URL https:// doi.org/10.18653/v1/p19-1472. Zhang, X., Xie, W., Li, Y ., Lei, J., and Du, Q. Filter pruning via learned representation median in the fre- quency domain.IEEE Trans. Cybern., 53(5):3165–3175,

    doi

  9. [2022]

    URL https: //doi.org/10.48550/arXiv.2210.17323

    doi: 10.48550/ARXIV .2210.17323. URL https: //doi.org/10.48550/arXiv.2210.17323. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Scho...

    doi arXiv

  10. [2023]

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D

    URL https://proceedings.mlr.press/ v202/frantar23a.html. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: accurate post-training quantization for genera- tive pre-trained transformers.CoRR, abs/2210.17323,

    arXiv

  11. [2024]

    URL https: //doi.org/10.1609/aaai.v38i10.28960

    doi: 10.1609/AAAI.V38I10.28960. URL https: //doi.org/10.1609/aaai.v38i10.28960. Bai, J., Bai, S., Chu, Y ., Cui, Z., Dang, K., Deng, X., Fan, Y ., Ge, W., Han, Y ., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X....

    doi arXiv

  12. [6399]

    v34i05.6399

    URL https://doi.org/10.1609/aaai. v34i05.6399. Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A sim- ple and effective pruning approach for large language models. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Ope...

    doi

  13. [8740]

    doi: 10.1609/AAAI.V34I05

    AAAI Press, 2020. doi: 10.1609/AAAI.V34I05

    2020 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.