DarwinLM: Evolutionary Structured Pruning of Large Language Models

Dan Alistarh; Eldar Kurtic; Oliver Sieberling; Shengkun Tang; Zhiqiang Shen

REVIEW 4 major objections 6 minor 5 cited by

DarwinLM: Evolutionary Structured Pruning of Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DarwinLM treats structured pruning as an evolutionary search, selects candidates by how well they recover under brief fine-tuning, and produces a 2.6B model that beats a baseline trained with five times more data.

desk verdict Solid pruning paper with a real headline result, but the training-aware selection at its core is only validated at a 2M-token horizon, not the 10B tokens used in the main experiments. read the letter →

arxiv 2502.07780 v4 pith:OMHV5D2B submitted 2025-02-11 cs.LG cs.CL

Shengkun Tang , Oliver Sieberling , Eldar Kurtic , Zhiqiang Shen , Dan Alistarh This is my paper · ORCID

classification cs.LGcs.CL

keywords structuredpruningevolutionarysearchlargelanguagemodelstraining-awareselectionnon-uniformsparsityallocationsecond-orderKL-divergencefitnesspost-trainingrecovery

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DarwinLM is a method for structured pruning of large language models that treats the search for the best non-uniform sparsity allocation as an evolutionary process. Starting from a uniformly pruned parent, it generates candidate offspring by shifting one sparsity step from one module to another, briefly fine-tunes each candidate on a tiny token budget, and keeps the survivors that look most recoverable. The paper's central claim is that this training-aware selection finds structures that recover better under continued training: a 2.6B-parameter pruned Llama-2-7B averages 62.8 across eight benchmarks, beating ShearedLlama-2.7B (62.6) while using 10B tokens of post-training data instead of 50B. A careful reader should care because structured pruning delivers real speedups on ordinary hardware, and the paper argues that the right search over sparsity structure is worth more than extra recovery data.

What carries the argument

The load-bearing machinery is a sparsity-level database combined with a sparsity-preserving mutation operator and a training-aware selection loop. The database stores, for every attention and MLP module, a set of pre-pruned versions at discrete sparsity levels, each produced by second-order structured pruning: an OBS-style computation that uses the inverse Hessian of the layer inputs to choose which heads or intermediate columns to remove and to update the remaining weights. A candidate model is stitched together by picking one sparsity level per module, and a level-switch mutation raises the sparsity of one module by one step while lowering another by the same step, so total sparsity is conserved and every offspring automatically satisfies the target constraint. Selection then fine-tunes all candidates on progressively larger small token budgets (10K, 50K, 100K, then 200K tokens) and keeps the top survivors (8, 4, 2, then 1) ranked by KL divergence between the sparse candidate's outputs and the dense model's outputs on calibration data. The claim that rests on this machinery is that KL divergence measured after a brief fine-tune ranks candidates in the same order as their performance after full 10B-token post-training.

What would settle it

Replicate the Llama-2-7B search with a larger offspring population, say 64 per generation, and measure the rank correlation between KL divergence after the 200K-token selection step and average benchmark accuracy after full 10B-token post-training. If the correlation is weak for larger populations, or fails to hold for Llama-3.1-8B or Qwen-2.5-14B-Instruct, the training-aware selection claim is not general. A cheaper check that also isolates the contribution of search: post-train the best one-shot-KL structure and the best training-aware structure under identical 10B-token recipes across several seeds; if the one-shot structure matches the evolved one, the central claim fails. The paper's own Table 4 runs this comparison only at 1B tokens, so the 10B-token version is an open experiment.

Watch

Extended reading notes

Core claim

DarwinLM's central claim is that the one-shot best sparse structure and the structure that performs best after continued training are different things, and that the latter can be identified before committing to expensive training. The paper shows that a lightweight multistep fine-tuning of offspring candidates, 10K, 50K, 100K, then 200K tokens, followed by ranking via KL divergence between dense and sparse outputs, picks out the candidates that recover best under full 10B-token post-training (Figure 2). With this selection, the evolved 2.6B Llama-2 model beats ShearedLlama in average downstream accuracy while using five times fewer tokens for recovery; the 4.6B Llama-3.1 model beats OLMO-7B, a dense model trained on roughly 250 times more data; and the 8.4B Qwen-2.5 model reaches 68.4 average after only 10B tokens of continued training. In the one-shot setting, the same search outscores ZipLM on every model tested, on Llama-2 at 2.7B parameters versus ZipLM's 4.0B, on Llama-3.1, and on Qwen-2.5, and far outscores uniform pruning at similar sizes.

Load-bearing premise

The load-bearing premise is that after a candidate is fine-tuned on just 10,000 to 200,000 tokens, its KL divergence from the dense model reliably predicts which candidate will be best after full 10B-token post-training; the paper demonstrates this correlation in a single 16-offspring experiment on Llama-2-7B (Figure 2).

Editorial extensions

If this is right

On Llama-2-7B, the 2.6B DarwinLM model reaches 62.8 average accuracy with 10B recovery tokens, above ShearedLlama's 62.6 with 50B tokens, so a better compressed model can be produced at one fifth of the recovery-training cost.
The 4.6B model pruned from Llama-3.1-8B scores 63.7 average, beating the 7B dense OLMO model (62.9) trained on roughly 2.5T tokens, which indicates that pruning a strong parent plus modest recovery can outperform much larger scratch-trained models.
The method transfers to instruction-tuned models: the 8.4B Qwen-2.5 variant reaches 68.4 average after only 10B tokens, versus 56.9 for uniform pruning at a similar size.
The one-shot results isolate the value of the search itself: non-uniform allocation scores 57.2 on Llama-2 at 2.7B parameters, where uniform pruning collapses to 40.1 at 3.4B parameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The selection mechanism is really a predictor of recoverability, so the same multistep fine-tuning and ranking recipe could be reused to choose among other post-training decisions, such as data mixtures, learning-rate schedules, or which structure to keep under a fixed training budget, not just sparsity allocations.
The 5x data-efficiency claim rests on a rank correlation shown in a single 16-offspring experiment (Figure 2); whether the correlation holds at larger population sizes and on Llama-3.1 and Qwen is untested, and is the most direct place to probe the method.
Because the mutation operator conserves total sparsity, the fitness function is swappable: replacing KL divergence with a latency or memory measurement would let the same evolutionary loop optimize sparsity for specific hardware targets rather than for accuracy alone.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

DarwinLM is a well-engineered structured pruning paper with one genuinely strong result: a 2.6B Llama-2 model that beats ShearedLlama-2.7B on average downstream accuracy after only 10B tokens of post-training versus 50B. The comparison against ShearedLlama retrained on the same 10B tokens is fair and the checkpoints are released, which puts the claim on solid ground. The method itself is a clean combination of second-order OBS-style pruning databases, an evolutionary search with sparsity-preserving level-switch mutations, and a multi-step training-aware selection stage. The level-switch mutation and the staged selection with increasing token budgets are new relative to ZipLM and EvoPress, and Figure 2 gives a clear motivation.

The soft spots are real but not fatal. The biggest one: the training-aware selection mechanism is argued with 'training on little data is a good predictor of more large-scale training,' but the only direct demonstration, Figure 2, compares selection steps against a 'full training' run of 2M tokens. The actual post-training in the main experiments is 10B tokens. The ablation in Table 4 goes to 1B tokens at most. So the core predictive claim is untested at the scale at which the headline result is evaluated. Rankings of offspring can plausibly reorder between 2M and 10B. I would not block the paper over this, because the main result may not depend critically on TAS, but the paper should either soften the claim or add re-ranking evidence from the 10B runs.

There are also two clear data errors. Table 3 lists identical numbers for Qwen-2.5-14B-Instruct and Llama-3.1-8B, which is a copy-paste mistake. And the calibration set is described as '16 sequences' in Section 4.1 but '2,048 sequences with 4,096 tokens' in Appendix A.1. Those need fixing. Also no error bars or multiple seeds; for a stochastic search that's a minor concern given the compute, but worth noting.

Overall, the paper is a solid contribution to LLM pruning. The empirical comparisons are mostly fair, the code and weights are released, and the method is clearly specified. It deserves a serious peer review. I'd suggest accepting it conditionally after the data errors are corrected and the TAS validation gap is addressed, either with new experiments or a more modest claim.

Referee Report

4 major / 6 minor

Summary. The paper proposes DarwinLM, a structured pruning method for LLMs that combines second-order layer-wise pruning with an evolutionary search. A sparsity-level database is built with OBS-style structured pruning, and offspring models are generated by sparsity-preserving level-switch mutations. A multi-step training-aware selection (TAS) process trains candidates on progressively larger token budgets (10K-200K) and selects survivors by KL divergence against the dense model. The main empirical claims are: on Llama-2-7B, DarwinLM at 2.6B parameters with 10B post-training tokens reaches 62.8 average downstream accuracy, surpassing ShearedLlama's 62.6 at 2.7B and 50B tokens; on Llama-3.1-8B (4.6B) and Qwen-2.5-14B-Instruct (8.4B) it outperforms ZipLM and matches or exceeds OLMo baselines; and in one-shot settings it beats uniform pruning, ZipLM, EvoPress, ShortGPT, and Shortened-Llama across sparsities. Code and weights are released.

Significance. If the claims hold, the contribution is practically significant: a hardware-agnostic structured pruning pipeline that matches or exceeds ShearedLlama with roughly 5x less post-training data, and a sensible integration of evolutionary search with training-aware selection. The release of code and weights is a clear strength for reproducibility. However, the current manuscript contains unresolved data inconsistencies and validates the core selection mechanism at a token scale far below the scale used in the headline experiments, so the significance cannot be fully assessed without revision.

major comments (4)

[§3.5, Figure 2, Table 4] The assumption that "training on little data is a good predictor of more large-scale training" is demonstrated only against a 2M-token "full training" in Figure 2 and a 1B-token ablation in Table 4, while the headline results in Tables 1 and 2 use 10B tokens. This is a 5000x gap between the validation and deployment horizons, and no evidence is provided that offspring rankings are stable from 2M to 10B tokens. Since TAS is the paper's core claimed novelty, this gap is load-bearing; please add a validation at the actual post-training scale (e.g., rank correlation between selection-step KL and final 10B-token accuracy, or at least a 2B/5B intermediate checkpoint) or explicitly narrow the claim to short-horizon prediction.
[Algorithm 1 / §3.5] The pseudocode does not implement generational evolution as described in the text. The mutation loop runs before the selection loop, and the variable `candidates` is reset to `[parent]` at each iteration of the outer loop, so selection is applied only to the offspring of the last generation. The text states that "The final surviving candidate is selected as the starting point for the next generation," which is not reflected in Algorithm 1. As written, the algorithm is ambiguous and not reproducible; please correct the pseudocode or the text so that the iterative parent-update process is unambiguous.
[Table 3, §4.1, Appendix A.1] The Qwen-2.5-14B-Instruct row in Table 3 lists PIQA 69.4, WG 57.3, ArcC 34.2, and HS 44.6, which are exactly the Llama-3.1-8B values from the same table and conflict with the Qwen one-shot row in Table 2 (PIQA 73.9, WG 60.5, ArcC 48.0, HS 53.3). This appears to be a copy-paste error and must be corrected. Additionally, the calibration data size is reported as "at most 16 sequences" in §4.1, "2048 calibration samples" and "2K calibration data" in the Implementation Details, and "2,048 sequences with 4,096 tokens" in Appendix A.1. These conflicting numbers need to be reconciled, since the lightweight nature of the search depends on which value is correct.
[§4.2 / Table 2] For Qwen-2.5-14B-Instruct, the text states that "After post-compression training with only 2B tokens, the performance of DarwinLM increases to 68.1," but Table 2 reports only a 10B-token row with an average of 68.4. The 2B result is neither shown in the table nor otherwise documented, so the claim cannot be verified. In addition, Tables 1 and 2 report single-run evaluations without error bars; given that the headline advantage over ShearedLlama is 0.2 average points (62.8 vs 62.6), variance estimates or multiple evaluation runs would strengthen the comparison.

minor comments (6)

[Figure 1 caption] The caption contains the typo "Offsping Generation"; it should be "Offspring Generation".
[§3.5, §4.1, Appendix A.1] The token budget "8196" should likely be "8192"; the number of selection steps and survivor counts are also inconsistent (3 steps with [8,4,1] in §3.5/Figure 2 versus 4 steps with [8,4,2,1] in §4.1 and Appendix A.1). Please make these details consistent across the paper.
[Appendix A.1] The text refers to "Qwen-2.5-8B-Instruct" when the experiments use Qwen-2.5-14B-Instruct; correct the model name.
[Conclusion] The conclusion states that prior methods required "10x more data and training computation," while the abstract and main text emphasize a 5x reduction compared to ShearedLlama; please clarify which comparison the 10x figure refers to.
[§3.5 / Figure 2] Calling 2M tokens a "large-scale full training" is misleading given that the paper's post-training uses 10B tokens; consider renaming it to "2M-token training" or similar to avoid confusion.
[Table 4] The TAS ablation reports only PIQA, SciQ, and ArcE without error bars; since this table supports a central methodological claim, please report the full benchmark suite or justify the subset and indicate whether the observed gaps are robust.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: DarwinLM's headline comparison to ShearedLlama is an external empirical benchmark, and the training-aware selection premise rests on a short-horizon empirical correlation, not on a fitted parameter or self-referential construction.

full rationale

The paper's derivation chain is not circular. The search optimizes an explicit KL-divergence fitness (Eq. 8) on calibration data; the final claim is downstream accuracy after 10B-token post-training, evaluated with the lm-evaluation-harness and compared against external baselines (ShearedLlama, ZipLM, OLMO). No fitted constant is later renamed as a prediction: the evolutionary search selects offspring by KL after 10K-200K token training, and the final model's accuracy is measured independently. The paper does rely on the assumption that small-scale training KL rankings predict large-scale training quality; its only direct evidence (Figure 2) uses 2M tokens as 'full training,' which is shorter than the 10B-token post-training used for the main result. This is a validation gap or correctness risk, not circularity, because the selection metric and the reported outcome are distinct and the comparison to ShearedLlama is external. The method borrows KL fitness and multi-step selection from EvoPress and second-order pruning from ZipLM, both from overlapping authors, but these are methodological building blocks rather than theorems invoked to forbid alternatives; the central result stands on the empirical comparison. Accordingly, no circular step meets the evidentiary bar.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method does not introduce new physical or mathematical entities. Its central claims rest on several domain assumptions about the fitness proxy and the predictive power of small-scale training, plus hand-chosen hyperparameters. The most load-bearing assumptions are the validity of the training-aware selection and the representativeness of Fineweb-Edu.

free parameters (5)

Number of sparsity levels N_l = 10
Determines the resolution of the sparsity level database and the step size for mutations. Chosen by hand, not fitted to data.
Target sparsity level = 5 and 6
The compression targets chosen for the experiments. Level 5 for Llama-3.1-8B and Qwen-2.5-14B-Instruct; level 6 for the final Llama-2-7B model after two-stage pruning.
Offspring selection token budgets = [10K, 50K, 100K, 200K] training tokens and [1024, 2048, 4096, 8192] evaluation tokens
Hand-chosen schedules for the multi-step training-aware selection. These are not derived from theory and could affect which structures survive.
Number of generations and offspring per generation = 200 generations, lambda=16; 500 generations for second-stage Llama-2
Search depth and population size chosen for compute efficiency. No evidence is given that the search has converged.
Calibration set size = Stated as both 'at most 16' and 2048 sequences
The paper is internally inconsistent about this value, which is concerning because the second-order pruning and fitness evaluation depend on it.

assumptions (4)

domain assumption The KL divergence between dense and sparse outputs on calibration data is a valid fitness proxy for final model quality.
Used throughout the evolutionary search (Section 3.4). The claim that training-aware selection helps relies on this proxy being informative.
domain assumption Training on 10K to 200K tokens predicts relative performance after training on 10B tokens.
Central to the training-aware selection; Figure 2 shows this for a single Llama-2-7B case, but it is assumed to generalize to other models and sparsity levels.
standard math The OBS second-order update (Eqs. 2-4) accurately measures the importance of each pruned structure.
This is the standard Optimal Brain Surgeon derivation from prior work (Hassibi & Stork, Kurtic et al.), and the paper inherits its assumptions, including the local quadratic approximation of the loss.
domain assumption Fineweb-Edu is a representative calibration and training distribution for the downstream tasks evaluated.
Calibration data, offspring training data and post-training data are all sampled from Fineweb-Edu. If this distribution is not representative, the reported benchmark numbers could be optimistic.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DarwinLM: Evolutionary Structured Pruning of Large Language Models." pith.science (2026). https://pith.science/paper/OMHV5D2B

@misc{pith2026250207780,
  author       = {Pith},
  title        = {Pith review of: DarwinLM: Evolutionary Structured Pruning of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OMHV5D2B}},
  note         = {Machine review of arXiv:2502.07780}
}

read the original abstract

Large Language Models (LLMs) have achieved significant success across various NLP tasks. However, their massive computational costs limit their widespread use, particularly in real-time applications. Structured pruning offers an effective solution by compressing models and directly providing end-to-end speed improvements, regardless of the hardware environment. Meanwhile, different components of the model exhibit varying sensitivities towards pruning, calling for non-uniform model compression. However, a pruning method should not only identify a capable substructure, but also account for post-compression training. To this end, we propose DarwinLM, a method for training-aware structured pruning. DarwinLM builds upon an evolutionary search process, generating multiple offspring models in each generation through mutation, and selecting the fittest for survival. To assess the effect of post-training, we incorporate a lightweight, multistep training process within the offspring population, progressively increasing the number of tokens and eliminating poorly performing models in each selection stage. We validate our method through extensive experiments on Llama-2-7B, Llama-3.1-8B and Qwen-2.5-14B-Instruct, achieving state-of-the-art performance for structured pruning. For instance, DarwinLM surpasses ShearedLlama while requiring 5x less training data during post-compression training. Code is at: https://github.com/IST-DASLab/DarwinLM

Figures

Figures reproduced from arXiv: 2502.07780 by the authors.

**Figure 1.** Visual illustration of DarwinLM pipeline. 1) generate sparsity level database with different sparsities by second-order structured pruning. 2) evolutionary search with training-aware selection based on the sparsity level database. Other Compression Methods. Several approaches have been explored in the literature to reduce computational and memory requirements of LLMs without significantly degrading performance, inc… view at source ↗

**Figure 2.** Motivation of the training-aware selection. The Y-axis depicts the KL-Divergence of the model after full post-training while x-axis is the KL-Divergence after small-scale data training. The results indicate that our training-aware selection can select the best offspring for large-scale training. 3.5. Evolutionary Search Algorithm We approach the problem in Equation (1) via evolutionary search. The algorithm, describ… view at source ↗

**Figure 3.** Performance comparison of DarwinLM and ShearedLlama with different training token numbers. DarwinLM achieves better performance than ShearedLlama on all training token number settings. structured pruning, particularly at high sparsity. After postcompression training, the pruned models see a significant recovery in performance. Notably, with only 10B tokens for training, DarwinLM reaches an average score of 62.8, … view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Comparison of DarwinLM and other one-shot methods that remove modules entirely. Our method consistently outperforms across all sparsity levels, demonstrating the effectiveness of our finer-grained structured pruning approach. Note that the y-axis is log-scaled. it onl…

**Figure 5.** Figure 5: Post-training comparison of ShearedLlama and DarwinLM on each benchmark. B. Limitations & Future Work While our non-uniform pruning of LLMs using a speed-aware evolutionary search algorithm offers several advantages, it also has certain limitations: 1) The training-awa…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Omega-S: A Functional Resilience Index for LLM Fine-Tuning
cs.LG 2026-08 conditional novelty 6.0 of 10

Omega-S, a penalty on node-degree variance in the weight matrix, improves code retention during LoRA fine-tuning of Llama-3-8B, while its advertised clustering/topological channel is inert.
Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation
cs.LG 2025-09 conditional novelty 6.0 of 10

PrunedLoRA tunes large-rank LoRA adapters and progressively structurally prunes A and B to a target low rank, empirically closing much of the LoRA-to-full-fine-tuning gap.
GeLaCo: An Evolutionary Approach to Layer Compression
cs.CL 2025-07 conditional novelty 6.0 of 10

Evolutionary search over layer-merging configurations, scored by module-wise activation similarity, yields competitive LLM compression and the first size-quality Pareto fronts.
SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot
cs.LG 2025-06 conditional novelty 6.0 of 10

SparseSSM extends OBS-style second-order pruning to Mamba's discretized, time-shared state-transition matrix, pruning 50% of its weights in one pass without fine-tuning.
AutoChemSchematic AI: Agentic Physics-Aware Automation for Chemical Manufacturing Scale-Up
cs.LG 2025-05 reject novelty 4.0 of 10

The framework trains small models on synthetic AI-generated data to produce PFD/PID text, then validates two examples by manual DWSIM setup, leaving the industrial-viability claim unproven.

Reference graph

Works this paper leans on

30 extracted references · 6 canonical work pages · cited by 5 Pith papers

[1]

Gqa: Training generalized multi-query transformer models from multi-head check- points

Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y ., Lebr´on, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.13245,

arXiv
[5]

Gptq: Accurate post-training quantization for generative pre- trained transformers

Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210.17323,

arXiv
[6]

org/records/10256836,

URL https://zenodo. org/records/10256836,

arXiv
[7]

H., Ivison, H., Magnusson, I., Wang, Y ., et al

Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y ., et al. Olmo: Accelerating the science of lan- guage models. arXiv preprint arXiv:2402.00838,

arXiv
[11]

Shortened llama: A simple depth pruning for large language models

Kim, B.-K., Kim, G., Kim, T.-H., Castells, T., Choi, S., Shin, J., and Song, H.-K. Shortened llama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.02834,

arXiv
[12]

Structural pruning of large language models via neural architecture search

Klein, A., Golebiowski, J., Ma, X., Perrone, V ., and Archam- beau, C. Structural pruning of large language models via neural architecture search. In AutoML Conference 2023,

work page 2023
[13]

The optimal bert surgeon: Scalable and accurate second-order pruning for large language models

Kurtic, E., Campos, D., Nguyen, T., Frantar, E., Kurtz, M., Fineran, B., Goin, M., and Alistarh, D. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint arXiv:2203.07259,

arXiv
[14]

Evolving knowledge distillation with large lan- guage models and active learning

Liu, C., Zhao, F., Kuang, K., Kang, Y ., Jiang, Z., Sun, C., and Wu, F. Evolving knowledge distillation with large lan- guage models and active learning. In Calzolari, N., Kan, M.-Y ., Hoste, V ., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation...

work page 2024

Show all 30 references

[15]

Logiqa: A challenge dataset for machine reading comprehension with logical reasoning

Liu, J., Cui, L., Liu, H., Huang, D., Wang, Y ., and Zhang, Y . Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124,

2007 arXiv
[16]

Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., and Wei, F

URL https://huggingface.co/datasets/ HuggingFaceFW/fineweb-edu. Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., and Wei, F. The era of 1-bit llms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764,

arXiv
[17]

Shortgpt: Layers in large language mod- els are more redundant than you expect

Men, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y ., Han, X., and Chen, W. Shortgpt: Layers in large language mod- els are more redundant than you expect. arXiv preprint arXiv:2403.03853,

arXiv
[18]

T., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P

Muralidharan, S., Sreenivas, S. T., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Compact language models via pruning and knowledge distillation. arXiv preprint arXiv:2407.14679,

arXiv
[20]

Evopress: Towards optimal dynamic model compression via evolutionary search.arXiv preprint arXiv:2410.14649,

Sieberling, O., Kuznedelev, D., Kurtic, E., and Alistarh, D. Evopress: Towards optimal dynamic model compression via evolutionary search.arXiv preprint arXiv:2410.14649,

arXiv
[21]

Bi-mamba: Towards accurate 1-bit state space models.arXiv preprint arXiv:2411.11843,

Tang, S., Ma, L., Li, H., Sun, M., and Shen, Z. Bi-mamba: Towards accurate 1-bit state space models.arXiv preprint arXiv:2411.11843,
[22]

Structured pruning for efficient gen- erative pre-trained language models

Tao, C., Hou, L., Bai, H., Wei, J., Jiang, X., Liu, Q., Luo, P., and Wong, N. Structured pruning for efficient gen- erative pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023 , pp. 10880–10895,

2023
[23]

Llama 2: Open foundation and fine- tuned chat models

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 ,

arXiv
[24]

Bitnet: Scaling 1- bit transformers for large language models.arXiv preprint arXiv:2310.11453,

Wang, H., Ma, S., Dong, L., Huang, S., Wang, H., Ma, L., Yang, F., Wang, R., Wu, Y ., and Wei, F. Bitnet: Scaling 1- bit transformers for large language models.arXiv preprint arXiv:2310.11453,

arXiv
[25]

Structured pruning of large language models

Wang, Z., Wohlwend, J., and Lei, T. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6151–6162,

2020
[27]

A survey on knowledge distillation of large language models

Xu, X., Li, M., Tao, C., Shen, T., Cheng, R., Li, J., Xu, C., Tao, D., and Zhou, T. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116, 2024a. 10 DarwinLM: Evolutionary Structured Pruning of Large Language Models Xu, Y ., Han, X., Yang...

arXiv
[28]

Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

1905 arXiv
[29]

Appendix A.1

11 DarwinLM: Evolutionary Structured Pruning of Large Language Models A. Appendix A.1. More Implementation Details. Details of Second-Order Structured Pruning. We utilize 2,048 sequences with 4,096 tokens from the Fineweb-Edu dataset as calibration data for Llama-2-7B, Llama-3...
[30]

Hyper-parameter details for post-training on DarwinLM-2.6B, DarwinLM-4.4B, and DarwinLM-8.4B. Parameter DarwinLM-2.6B DarwinLM-4.4B DarwinLM-8.4B Learning rate 1e-4 1e-4 1e-4 Global batch size 1,024 1152 2048 Warm-up steps 50 steps 10 steps 50 steps LR decay scheduler Cosine C...
[1992]

Measuring mas- sive multitask language understanding

Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300,

2009 arXiv
[2015]

Distill- ing step-by-step! outperforming larger language models with less training data and smaller model sizes

Hsieh, C.-Y ., Li, C.-L., Yeh, C.-K., Nakhost, H., Fujii, Y ., Ratner, A., Krishna, R., Lee, C.-Y ., and Pfister, T. Distill- ing step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301,

arXiv
[2018]

Spqr: A sparse-quantized representation for near-lossless llm weight compression

Dettmers, T., Svirschevski, R., Egiazarian, V ., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078,

arXiv
[2019]

Think you have solved question answering? try arc, the ai2 reasoning challenge

Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

arXiv
[2020]

F., and Gardner, M

Welbl, J., Liu, N. F., and Gardner, M. Crowdsourc- ing multiple choice science questions. arXiv preprint arXiv:1707.06209,

arXiv
[2021]

Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter

Sanh, V . Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108,

1910 arXiv
[2023]

The llama 3 herd of models

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

arXiv
[2024]

Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., and Roberts, D. A. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887,

arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Gqa: Training generalized multi-query transformer models from multi-head check- points

Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y ., Lebr´on, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.13245,

arXiv

[2] [5]

Gptq: Accurate post-training quantization for generative pre- trained transformers

Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210.17323,

arXiv

[3] [6]

org/records/10256836,

URL https://zenodo. org/records/10256836,

arXiv

[4] [7]

H., Ivison, H., Magnusson, I., Wang, Y ., et al

Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y ., et al. Olmo: Accelerating the science of lan- guage models. arXiv preprint arXiv:2402.00838,

arXiv

[5] [11]

Shortened llama: A simple depth pruning for large language models

Kim, B.-K., Kim, G., Kim, T.-H., Castells, T., Choi, S., Shin, J., and Song, H.-K. Shortened llama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.02834,

arXiv

[6] [12]

Structural pruning of large language models via neural architecture search

Klein, A., Golebiowski, J., Ma, X., Perrone, V ., and Archam- beau, C. Structural pruning of large language models via neural architecture search. In AutoML Conference 2023,

work page 2023

[7] [13]

The optimal bert surgeon: Scalable and accurate second-order pruning for large language models

Kurtic, E., Campos, D., Nguyen, T., Frantar, E., Kurtz, M., Fineran, B., Goin, M., and Alistarh, D. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint arXiv:2203.07259,

arXiv

[8] [14]

Evolving knowledge distillation with large lan- guage models and active learning

Liu, C., Zhao, F., Kuang, K., Kang, Y ., Jiang, Z., Sun, C., and Wu, F. Evolving knowledge distillation with large lan- guage models and active learning. In Calzolari, N., Kan, M.-Y ., Hoste, V ., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation...

work page 2024