Pith. sign in

REVIEW 3 major objections 4 minor 4 cited by

EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that sub-billion language models can outperform direct-pretrained baselines by pruning a larger pretrained model during continued pretraining, retaining its capabilities instead of learning them from scratch.

desk verdict A genuinely new scaling recipe for pruning-aware pretraining, but the EfficientLLM flagship results are clouded by unfair token accounting versus the source teacher models. read the letter →

arxiv 2502.06663 v2 pith:P2OINC3D submitted 2025-02-10 cs.LG

Xingrun Xing , Zheng Liu , Shitao Xiao , Boyan Gao , Yiming Liang , Wanpeng Zhang , Haokun Lin , Guoqi Li
show 1 more author
Jiajun Zhang
This is my paper · ORCID
classification cs.LG
keywords pruning-awarepretrainingedgelanguagemodelsstructuralpruningsaliency-drivenarchitecturesearchLLMcompressionsub-billionbi-leveloptimizationTaylorexpansion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that small language models for edge devices need not be trained from scratch; instead, they can be grown down from an already-trained larger model by interleaving structural pruning with pretraining on large-scale data. The authors propose pruning-aware pretraining, in which a source model is continuously pruned along saliency-guided minimal parameter groups while gradients continue to update the surviving weights, so the final small model retains capabilities of the large source rather than paying the full cost of learning them from random initialization. They introduce the EfficientLLM family (134M, 469M, and 1.1B parameters) and report that it outperforms direct-pretrained baselines in the 100M-1B range, including SmolLM, Qwen2.5-0.5B, OLMo-1B, Llama3.2-1B, and MobileLLM on common-sense reasoning benchmarks. If correct, this would break the usual scaling-law trade-off for small models: data-efficient edge models could be produced by compression-aware pretraining instead of ever-larger token budgets.

What carries the argument

The load-bearing mechanism is structural pruning-under-pretraining formulated as bi-level optimization over minimal parameter groups. Three pruning types define the space: per-head pruning in self-attention, per-channel pruning in FFN intermediate channels, and per-channel pruning of the transformer stem (embedding, layer inputs/outputs, LM head), each group coupled across all layers. At every step the algorithm computes a second-order Taylor saliency (with a global diagonal Hessian for detection and a layerwise $XX^\top$ approximation for weight updating), picks the mini-group with minimal saliency among $\{S_{\text{attn}}, S_{\text{ffn}}, S_{\text{stem}}\}$, prunes those coupled channels, then runs one gradient step; alternating pruning and gradient descent is called pruning-aware pretraining $\times 1$. This turns architecture search into repeated saliency-based channel removal and makes the final architecture agnostic to human design.

What would settle it

Train EfficientLLM-A-469M exactly as described, then pretrain an identical 469M architecture from random initialization on the same total data, including the source model's original pretraining corpus plus the pruning and continued tokens, and compare on the same benchmark harness; if the scratch model matches or beats EfficientLLM, the retained-performance mechanism is not what drives the gains. A cheaper check is to rerun the pipeline after randomly shuffling the source model's weights while keeping its architecture, then prune and train; if accuracy stays the same, the source model's learned weights were irrelevant.

Watch

Extended reading notes

Core claim

The central claim is that scaling up pruning into pretraining lets sub-billion-parameter LLMs exceed the performance of state-of-the-art human-designed models trained by direct pretraining from scratch. Concretely, EfficientLLM-A-469M trained with 50B continued tokens averages 54.77 on seven common-sense tasks, beating SmolLM-360M (600B tokens) and Qwen2.5-0.5B (17T tokens); EfficientLLM-A-1.1B averages 60.75 with 320B continued tokens, beating OLMo-1B, TinyLlama-1.1B, and Llama3.2-1B. The paper explains the gap by saying the pruned model inherits the optimized behavior of SmolLM-1.7B, losing only 1.05 average accuracy in the 1.1B case, whereas direct pretraining must induce those behaviors from random weights. The authors also claim that saliency-driven search automatically discovers competitive architectures without manual design, such as hidden size 2048, FFN intermediate 4870, and 24 layers for the 1.1B model.

Load-bearing premise

The reported token counts leave out the original pretraining of the SmolLM source models; if that compute is charged to the pipeline, the claim that pruning-aware pretraining beats direct pretraining's cost-performance trade-off is not established.

Editorial extensions

If this is right

  • EfficientLLM at 134M, 469M, and 1.1B reports higher common-sense averages than direct-pretrained baselines of comparable or larger size, even when using far fewer tokens for the pruning-aware stage.
  • Scaling up the pruning stage itself, for example from 0.66B to 10.56B tokens when pruning Llama2-7B, improves accuracy at every measured pruning ratio, so pruning-data scale matters more than the choice of pruning metric.
  • Existing post-training pruning metrics, namely LLM-Pruner and SparseGPT-style second-order updates, can be promoted into pretraining and remain effective at pruning ratios above 70 percent.
  • Auto-designed architectures from saliency-driven pruning are competitive with human-designed tiny-model architectures, suggesting that manual depth and width search can be replaced by pruning dynamics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A strict cost accounting that includes the source model's original pretraining (for example SmolLM-360M's 600B tokens or SmolLM-1.7B's pretraining) would shrink the apparent data-efficiency advantage, because those tokens are not charged to EfficientLLM in the paper's tables.
  • If the mechanism generalizes, it should be testable on other source models and data mixtures: pruning-aware pretraining from a larger model at high ratios should beat scratch-trained small models on code, math, and long-context benchmarks, not only on common-sense QA.
  • The same alternating pruning-and-gradient loop could be extended to quantization-aware or distillation-aware pretraining, letting the final edge model inherit more than parameter structure from its larger source.
  • The saliency-driven search suggests a possible automatic way to choose model shape under a compute budget, which could remove the need for human architecture search when deploying language models at a given size.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes pruning-aware pretraining: start from a pretrained larger LLM (SmolLM-360M or SmolLM-1.7B), alternately run gradient descent and saliency-based structured pruning of minimal parameter groups, then perform continued pretraining on the pruned model. This produces a family of 134M–1.1B models called EfficientLLM. The authors report that these models outperform direct-pretrained baselines such as SmolLM, Qwen2.5-0.5B, OLMo-1B, TinyLlama, and Llama3.2-1B on common-sense benchmarks, and they claim this 'exceeds the traditional LLM scaling law.' They also show that scaling up existing post-training pruning metrics (LLM-Pruner, SparseGPT/OBC) to 5B–10B tokens improves large-ratio pruning.

Significance. If the central claim were established, the work would be significant: it would offer a data-efficient route to sub-billion-parameter edge models by retaining knowledge from larger pretrained models, and it would suggest that interleaved pruning with gradient descent is preferable to one-shot post-training pruning. The evaluation is broad, uses standard harnesses, and reports architectures and hyperparameters; the commitment to open sourcing models and code is a concrete strength. However, the headline claims of 'exceeding the scaling law' and 'data-efficient pretraining' are confounded by initialization from heavily pretrained teachers, and the paper does not provide the control experiments needed to separate the effect of warm-starting from the effect of the proposed interleaved pruning procedure. The significance is therefore conditional on additional experiments that charge the teacher's pretraining cost or otherwise isolate the mechanism.

major comments (3)
  1. [§4.1, Table 2, §4.2] The token accounting is asymmetric and undermines the central 'exceeds the scaling law' claim. Table 2's '#Tokens' column counts only continued pretraining for EfficientLLM (as the table footnote states), while EfficientLLM-469M and EfficientLLM-1.1B are initialized from SmolLM-1.7B, whose original pretraining (roughly 1T tokens on the SmolLM corpus) enters at zero cost. The comparison in §4.2, where 'EfficientLLM-469M with 50B continued pretraining tokens exceeds SmolLM-360M with 600B tokens,' is a comparison on newly spent tokens only, not on total training cost. Appendix Table 8 shows that after the full pipeline the best EfficientLLM-1.1B (60.75 average) remains below its own teacher SmolLM-1.7B (61.80), so the reported advantages over OLMo-1B, TinyLlama, and Llama3.2-1B are consistent with a compressed-teacher effect rather than with pruning-aware pretraining beating direct pretraining at matched total cost. A from-scratch control with the same final architecture, data, and total compute, or a clear statement of total training FLOPs including the teacher, is needed before the 'exceeds the scaling law' conclusion can be drawn.
  2. [§3.2, Eq. (3), §4.2, Fig. 2] The paper does not isolate the contribution of interleaving pruning steps with gradient descent (the methodological novelty in Eq. 3) from the contribution of merely inheriting a good initialization. The natural control is a one-shot post-hoc prune (e.g., LLM-Pruner) of SmolLM-1.7B to the same target shapes in Table 5, followed by the identical 320B–500B continued pretraining, without any intermediate pruning during gradient descent. Without this control, the reported gains over post-training pruning in Fig. 2 and Table 3 could be attributed to the continued pretraining stage or to the source model's knowledge, not to the proposed alternating optimization. This is a load-bearing experiment for the paper's central technical claim.
  3. [§4.2] The statement that 'with the same training data, EfficientLLM-469M with 50B continued pretraining tokens exceeds SmolLM-360M with 600B tokens' is misleading. EfficientLLM-469M is initialized from SmolLM-1.7B, not from SmolLM-360M, and the data composition is described as 'similar to SmolLM' rather than identical to the SmolLM-360M training data. The comparison therefore confounds model initialization, data composition, and token budget; it does not demonstrate that pruning-aware pretraining outperforms direct pretraining under matched conditions. Please either run the matched from-scratch control or rephrase the claim to describe an inheritance/compression result rather than a scaling-law violation.
minor comments (4)
  1. [Table 7] The continued-pretraining token counts for EfficientLLM-1.1B are inconsistent: Table 7 lists '50B/500B' while the main text (§4.1) and Table 2 use 320B. Please reconcile these numbers.
  2. [§4.1] The text says 'For EfficientLLM-134M, 460M, and 1.1B' but the model is referred to elsewhere as EfficientLLM-469M; please fix the inconsistency.
  3. [Throughout] There are numerous typos and misspellings, including 'Hession' for Hessian, 'acheve' for achieve, 'appilications' for applications, 'condersiding' for considering, 'fomulate' for formulate, 'delate' for delete, 'donates' for denotes, and 'similiarily' for similarly. A careful proofread is needed.
  4. [Table 3] The table footnote says the 50% pruning results use lm-evaluation-harness version 0.3.0 while other results use 0.4.3; this is justified in Appendix B.2, but it would be clearer to report all EfficientLLM numbers under the same harness version or to state the version in the table caption directly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: saliency-based pruning is an independent heuristic whose outputs are evaluated against external benchmarks, so no claimed prediction reduces to its inputs by construction.

full rationale

The paper's central derivation is Eq. 9, where a pruning mini-group is chosen by a second-order Taylor approximation of the pretraining loss, following the external LLM-Pruner heuristic. This saliency computation uses gradients and Hessian approximations from the current model state; it does not fit any parameter to the benchmark scores reported in Table 2 or Figures 4-6. The reported architecture 'auto-design' is the cumulative result of iteratively applying Eq. 10 to drop the currently lowest-saliency group, and the resulting models are then independently evaluated on ARC, BoolQ, HellaSwag, OBQA, PIQA, WinoGrande, and MMLU. None of these benchmark numbers is derived from a fitted parameter, and no 'prediction' is statistically forced by an earlier fit. The only citation with possible author overlap (Zhang et al. 2024b, Plug-and-Play, cited in the LLM pruning related-work sentence) is background context and is not load-bearing for any result. The main legitimate concern is benchmark fairness rather than circularity: Table 2 counts only pruning-aware and continued-pretraining tokens for EfficientLLM, while the SmolLM-360M and SmolLM-1.7B starting points enter at zero token cost, and Appendix Table 8 shows the best EfficientLLM-1.1B (60.75) remains below its teacher SmolLM-1.7B (61.80). That asymmetry means the headline 'exceeds the scaling law' comparison may partly reflect inherited teacher performance, but this is an evaluation-design issue, not a case where the paper's equations reduce to their own inputs. Accordingly, no circular step meeting the evidence bar is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the validity of Taylor-based saliency ranking during training and on the framing that the source model's pretraining is free. Neither is independently verified. No new entities are introduced; the main free parameters are training schedules and target sizes.

free parameters (4)
  • Pruning-to-gradient ratio = 4:1, 2:1, 1:1, 1:9
    Chosen by the authors to trade pruning speed against training; it determines the token count needed to reach the target size and affects final accuracy in Figure 5.
  • Continued pretraining token budget = 50B, 320B, 500B
    Selected per model size; the 500B variant gives the headline results, while the 50B variant is still reported as competitive.
  • Pruning-aware pretraining token budget = 50.3B, 72.1B, 36.7B
    Reported as determined by the number of pruning iterations to reach target sizes, but effectively a schedule choice that affects all results.
  • Target model sizes = 134M, 469M, 1.1B
    Chosen to match baseline sizes; the pruning schedule runs until these sizes are reached, so the model family definition depends on this choice.
assumptions (5)
  • domain assumption Taylor expansion of the loss around optimal weights approximates the global loss for pruning decisions.
    The method selects groups by saliency derived from Eq. 8; if this approximation fails under training dynamics, pruning choices may be wrong.
  • domain assumption The diagonal Hessian approximation used by LLM-Pruner captures global saliency.
    EfficientLLM-A inherits this approximation in Section 3.2; no independent validation is provided.
  • domain assumption Layer-wise Hessian approximation H = X X^T gives valid weight updates for remaining weights.
    Used for second-order updating in Section 3.3, following SparseGPT and OBC; the paper does not quantify its error during pretraining.
  • domain assumption Pruning all inputs or all outputs of a DAG node is sufficient to prune the entire network.
    Invoked in Section 3.2 to restrict saliency calculation to the output layer group; this reduces cost but is a structural heuristic.
  • domain assumption Pretraining data distribution similar to the source model preserves and improves the source model's capabilities.
    Section 4.1 sets the data composition to mimic SmolLM; if this match fails, the retained-performance claim weakens.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models." pith.science (2026). https://pith.science/paper/P2OINC3D

@misc{pith2026250206663,
  author       = {Pith},
  title        = {Pith review of: EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P2OINC3D}},
  note         = {Machine review of arXiv:2502.06663}
}
abstract

Modern large language models (LLMs) driven by scaling laws, achieve intelligence emergency in large model sizes. Recently, the increasing concerns about cloud costs, latency, and privacy make it an urgent requirement to develop compact edge language models. Distinguished from direct pretraining that bounded by the scaling law, this work proposes the pruning-aware pretraining, focusing on retaining performance of much larger optimized models. It features following characteristics: 1) Data-scalable: we introduce minimal parameter groups in LLM and continuously optimize structural pruning, extending post-training pruning methods like LLM-Pruner and SparseGPT into the pretraining phase. 2) Architecture-agnostic: the LLM architecture is auto-designed using saliency-driven pruning, which is the first time to exceed SoTA human-designed LLMs in modern pretraining. We reveal that it achieves top-quality edge language models, termed EfficientLLM, by scaling up LLM compression and extending its boundary. EfficientLLM significantly outperforms SoTA baselines with $100M \sim 1B$ parameters, such as MobileLLM, SmolLM, Qwen2.5-0.5B, OLMo-1B, Llama3.2-1B in common sense benchmarks. As the first attempt, EfficientLLM bridges the performance gap between traditional LLM compression and direct pretraining methods, and we will fully open source at https://github.com/Xingrun-Xing2/EfficientLLM.

Figures

Figures reproduced from arXiv: 2502.06663 by the authors.

Figure 1
Figure 1. An overview of pruning-aware pretraining. (a) Training loop includes the joint saliency detection and weight optimizing, pruning type selection from pruning space, and second-order weight updating. (b) Traditional post-training pruning can be embeded in the training loop to scale up. (c) Continuous model size compression in pretraining. for top-quality edge language models. Recently, ShearedL￾lama (Xia et al., 2023)… view at source ↗
Figure 2
Figure 2. Performance of Pruning-Aware Pretraining. By scaling up LLM-Pruner in pretraining, performance of the source model is retained even if the pruning rate more than 70%. exceeds SoTA methods without bells and whistles. • We explore the auto-designd architectures in modern pretraining for the first time. Saliency-driven architec￾tures are competitive with human best practices. 2. Preliminary and Related Works Edge Langu… view at source ↗
Figure 3
Figure 3. Three basic pruning typies in the pruning space. We plot all the weight metrics with shape [Dinput, Doutput]. In backpropagation (in orange), the saliency of the output layer group (in blue) is calculated according to Eq. 9. We first assumpt an optimal g ∗ t in each pruning step with respect to the pretraining loss, and solve how to acquire g ∗ t in the next subsection. In each pruning step, an op￾timal mini-group o… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Win rate of EfficientLLM in the instruction tuning task. training tokens like Qwen (Yang et al., 2024a;b), OLMo (Groeneveld et al., 2024), TinyLlama (Zhang et al., 2024a), EfficientLLM tries to retain the performance of larger and stronger models, which is significantl…
Figure 6
Figure 6. Figure 6: Generalization of pruning-aware pretraining for different pruning metrics. LLM-Pruner and SparseGPT are scaled up. in a iteration, respectively. When the target model size is reached, the pruning-aware pretraining requires 2.5B, 4.5B, 8.4B, and 72.1B tokens of pretrain…
Figure 7
Figure 7. Figure 7: Visualization of pruning-aware pretraining. We plot the saliency of the three pruning types and their pruning ratio in training. A.2. Architecture Comparisons [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Group Query Attention (GQA) pruning. In the case of GQA, cluster attention can be obtained through pruning. After pruning, the number of query heads is the same in each layer, and the cluster attention compresses the KV Cache. Pruning-aware pretraining could structural…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EGGS-PTP: An Expander-Graph Guided Structured Post-training Pruning Method for Large Language Models

    cs.LG 2025-08 conditional novelty 4.0 of 10

    EGGS-PTP adds a connectivity-preserving diagonal selection to RIA-style importance pruning, achieving slightly better perplexity under N:M sparsity.

  2. Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges

    cs.DC 2025-07 conditional novelty 4.0 of 10

    A survey that builds a taxonomy of edge-cloud LLM-SLM collaboration for inference and training, claiming to be the first to unify both phases.

  3. PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models

    cs.CV 2025-06 conditional novelty 4.0 of 10

    PAROAttention permutes tokens along frame, height, and width axes to make visual attention block-wise, enabling sparse and INT8/INT4 quantized attention with near-baseline generation quality.

  4. Enabling On-Device Medical AI Assistants via Input-Driven Saliency Adaptation

    cs.CL 2025-06 reject novelty 4.0 of 10

    A saliency-based pruning plus 4-bit quantization pipeline runs Gemma 7B and LLaMA 8B on edge hardware, but medical QA accuracy drops by up to 27 points, contradicting the 'minimal accuracy loss' claim.

Reference graph

Works this paper leans on

41 extracted references · 9 canonical work pages · cited by 4 Pith papers

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [5]

    Architecture comparisons between EfficientLLM and human-designd models. Model Hidden Size FFN Intermediate Attention Heads Head Dim Layer MobileLLM-125M 576 1536 9 64 30 EfficientLLM-A-134M 757 966 5 64 32 MobileLLM-350M 960 2560 15 64 32 Qwen2/2.5 896 4864 14 64 24 EfficientLLM-A-469M 1195 3006 19 64 24 MobileLLM-1B 1280 3584 20 64 54 ShearedLlama-1.3B 2...

    work page

  3. [6]

    Once- for-all: Train one network and specialize it for efficient deployment

    Cai, H., Gan, C., Wang, T., Zhang, Z., and Han, S. Once- for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791,

    arXiv 1908

  4. [7]

    Hyper-parameters in pruning-aware pretraining and continued pretraining stages. Model #Tokens Learning Rate WarmUp Steps Batchsize Text Length #GPU Pruning-134M 50.3B 2 × 10−3 500 2 M 2048 32 Continued Pretrain-134M 500B 2 × 10−3 10000 1 M 2048 32 Pruning-469M 72.1B 5 × 10−4 500 1 M 2048 32 Continued Pretrain-469M 50B/500B 2 × 10−3 10000 1 M 2048 40 Pruni...

    work page 2024

  5. [8]

    Choukse, E., Erez, M., and Alameldeen, A. R. Compresso: Pragmatic main memory compression. In 2018 51st An- nual IEEE/ACM International Symposium on Microarchi- tecture (MICRO), pp. 546–558. IEEE,

    work page 2018

  6. [10]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    arXiv

  7. [11]

    Pruner-zero: Evolving symbolic pruning metric from scratch for large language models

    Dong, P., Li, L., Tang, Z., Liu, X., Pan, X., Wang, Q., and Chu, X. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. arXiv preprint arXiv:2406.02924,

    arXiv

  8. [12]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, 9 EfficientLLM A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

Show all 41 references
  1. [13]

    Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A

    URL https://zenodo.org/records/ 12608602. Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y ., et al. Olmo: Accelerating the science of lan- guage models. arXiv preprint arXiv:2402.00838,

    arXiv

  2. [14]

    Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

  3. [17]

    W., Shao, Y

    Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y . S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length llm in- ference with kv cache quantization. arXiv preprint arXiv:2401.18079,

    arXiv

  4. [18]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  5. [19]

    Distillm: Towards streamlined distillation for large language models

    Ko, J., Kim, S., Chen, T., and Yun, S.-Y . Distillm: Towards streamlined distillation for large language models. arXiv preprint arXiv:2402.03898,

    arXiv

  6. [20]

    org/abs/2406.11794

    URL https://arxiv. org/abs/2406.11794. Li, S., Chen, J., Han, X., and Bai, J. Nuteprune: Efficient progressive pruning with numerous teachers for large lan- guage models. arXiv preprint arXiv:2402.09773,

    arXiv

  7. [21]

    Darts: Differentiable architecture search

    Liu, H., Simonyan, K., and Yang, Y . Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 ,

    arXiv

  8. [22]

    Mobilellm: Optimizing sub-billion parameter lan- guage models for on-device use cases

    Liu, Z., Zhao, C., Iandola, F., Lai, C., Tian, Y ., Fedorov, I., Xiong, Y ., Chang, E., Shi, Y ., Krishnamoorthi, R., et al. Mobilellm: Optimizing sub-billion parameter lan- guage models for on-device use cases. arXiv preprint arXiv:2402.14905,

    arXiv

  9. [23]

    Lv, B., Zhou, Q., Ding, X., Wang, Y ., and Ma, Z

    URL https://huggingface.co/ datasets/HuggingFaceFW/fineweb-edu. Lv, B., Zhou, Q., Ding, X., Wang, Y ., and Ma, Z. Kvpruner: Structural pruning for faster and memory-efficient large language models. arXiv preprint arXiv:2409.11057 ,

    arXiv

  10. [24]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789,

    arXiv

  11. [26]

    Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695,

    arXiv

  12. [27]

    Rethinking optimization and architecture for tiny language models

    Tang, Y ., Liu, F., Ni, Y ., Tian, Y ., Bai, Z., Hu, Y .-Q., Liu, S., Jui, S., Han, K., and Wang, Y . Rethinking optimization and architecture for tiny language models. arXiv preprint arXiv:2402.02791,

    arXiv

  13. [28]

    Llama: Open and efficient foundation lan- guage models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13971, 2023a. Touvron, H., Martin, L., Stone, K., Albert, P...

    arXiv

  14. [29]

    Redpajama: an open dataset for training large lan- guage models

    Weber, M., Fu, D., Anthony, Q., Oren, Y ., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V ., et al. Redpajama: an open dataset for training large lan- guage models. arXiv preprint arXiv:2411.12372,

    arXiv

  15. [30]

    Sheared llama: Accelerating language model pre-training via structured pruning

    Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694,

    arXiv

  16. [31]

    Nas-bert: Task-agnostic and adaptive-size bert com- pression with neural architecture search

    Xu, J., Tan, X., Luo, R., Song, K., Li, J., Qin, T., and Liu, T.-Y . Nas-bert: Task-agnostic and adaptive-size bert com- pression with neural architecture search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1933–1943,

    1933

  17. [32]

    Qwen2 techni- cal report, 2024a

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 techni- cal report, 2024a. URL https://arxiv.org/abs/ 2407.10671. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Q...

    2020 arXiv

  18. [33]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    1905 arXiv

  19. [34]

    Loraprune: Pruning meets low- rank parameter-efficient fine-tuning

    Zhang, M., Chen, H., Shen, C., Yang, Z., Ou, L., Yu, X., and Zhuang, B. Loraprune: Pruning meets low- rank parameter-efficient fine-tuning. arXiv preprint arXiv:2305.18403, 2023a. Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama: An open-source small language model. arXiv p...

    arXiv

  20. [35]

    org/abs/2205.01068, 3:19–0, 2023b

    URL https://arxiv. org/abs/2205.01068, 3:19–0, 2023b. Zhang, Y ., Bai, H., Lin, H., Zhao, J., Hou, L., and Can- nistraci, C. V . Plug-and-play: An efficient post-training pruning method for large language models. InThe Twelfth International Conference on Learning Representatio...

    arXiv

  21. [38]

    The OBD only uses the second-order term in Eq.9, which applied the diagonal of the Hessian matrix for approximate calculation

    and OBD (LeCun et al., 1989). The OBD only uses the second-order term in Eq.9, which applied the diagonal of the Hessian matrix for approximate calculation. A.3. Cluster Attention Key Value QueryGroup Query AttentionCluster AttentionKV Cache Compression Figure

    1989

  22. [39]

    In the case of GQA, cluster attention can be obtained through pruning

    Group Query Attention (GQA) pruning. In the case of GQA, cluster attention can be obtained through pruning. After pruning, the number of query heads is the same in each layer, and the cluster attention compresses the KV Cache. Pruning-aware pretraining could structurally prune...

    2023

  23. [41]

    • Common Sense Reasoning: Follow most of recent works (Xia et al., 2023; Ma et al., 2023; Li et al., 2024), we apply the widely used lm-evaluation-harness package (Gao et al.,

    to evaluate MMLU zero-shot performance. • Common Sense Reasoning: Follow most of recent works (Xia et al., 2023; Ma et al., 2023; Li et al., 2024), we apply the widely used lm-evaluation-harness package (Gao et al.,

    2023

  24. [1989]

    In EfficientLLM, the pruning ratio of 13 EfficientLLM hidden-size is smaller than attention heads and FFN intermediate channels driven by saliency

    1963 1542 12 64 24 As shown in Table 5, we compare the auto-designed architectures by saliency via pruning and the best practices of human design, including MobileLLM and Qwen2/2.5-0.5B, OLMo-1B, ShearedLlama-1.3B. In EfficientLLM, the pruning ratio of 13 EfficientLLM hidden-s...

    1963

  25. [1993]

    Measuring mas- sive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300,

    2009 arXiv

  26. [2018]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044,

    1905 arXiv

  27. [2019]

    Lorashear: Efficient large language model struc- tured pruning and knowledge recovery

    Chen, T., Ding, T., Yadav, B., Zharkov, I., and Liang, L. Lorashear: Efficient large language model struc- tured pruning and knowledge recovery. arXiv preprint arXiv:2310.18356,

    arXiv

  28. [2020]

    Gpt-neox-20b: An open-source autoregressive language model

    Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745,

    arXiv

  29. [2021]

    T., Muralidharan, S., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P

    Sreenivas, S. T., Muralidharan, S., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Llm pruning and distillation in practice: The minitron approach. arXiv preprint arXiv:2408.11796,

    arXiv

  30. [2022]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

    1901

  31. [2023]

    Gqa: Training generalized multi-query transformer models from multi-head check- points

    Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y ., Lebr´on, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.13245,

    arXiv

  32. [2024]

    L., Nascimento, M

    Ashkboos, S., Croci, M. L., Nascimento, M. G. d., Hoefler, T., and Hensman, J. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024, 2024a. Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D...

    arXiv

  33. [2025]

    Han, S., Mao, H., and Dally, W. J. Deep compres- sion: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149,

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.