Pith. sign in

REVIEW 3 major objections 5 minor 27 references

Activation outliers, not gradients, guide better LLM pruning

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 06:59 UTC pith:3FX6WNJ5

load-bearing objection PALS: activation-percentile layer sparsity for Wanda — solid LLaMA-2 result, honest about where it fails, but ablations underpowered the 3 major comments →

arxiv 2607.07557 v1 pith:3FX6WNJ5 submitted 2026-07-08 cs.CL cs.LG

PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning

Yazdan Jamshidi , Alexey Shvets This is my paper
classification cs.CL cs.LG
keywords LLM pruninglayer-wise sparsityactivation outliersone-shot pruningWandagradient-based pruningmodel compressiontransformer layers
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that one-shot LLM pruning should not treat all transformer layers equally. Standard methods like Wanda and SparseGPT apply the same sparsity ratio to every layer, ignoring that some layers carry more critical information than others. PALS (Percentile-Aware Layerwise Sparsity) computes the 99th percentile of activation magnitudes for each layer as a proxy for importance, then shifts each layer's sparsity target up or down within a ±5% band around the global target. Layers with large activation outliers — known to be critical for model function — get pruned less; layers with uniformly small activations get pruned more. On LLaMA-2-7B at 50% sparsity, this reduces WikiText-2 perplexity from 12.92 to 10.96 (mean over 9 runs, p<0.001). The benefit is architecture-dependent: LLaMA-3-8B gains marginally and Mistral-7B gains nothing, which the authors attribute to newer models distributing computation more uniformly across layers. A secondary and arguably more provocative finding is that gradient-based layer importance metrics — the seemingly principled approach — produce sparsity allocations worse than random, suggesting that gradient magnitude does not predict the damage caused by removing a large fraction of weights.

Core claim

The central object is the 99th percentile of activation magnitudes as a layer importance signal. The paper discovers that this tail statistic, when used to reallocate sparsity budgets across transformer layers within tight bounds (±5%), yields substantial perplexity improvements on LLaMA-2-7B while gradient-based importance metrics fail catastrophically. The failure of gradients is particularly striking: gradient-norm allocation produces 47.3 perplexity versus 12.9 for uniform allocation and 24.4 for random allocation, indicating that the relationship between infinitesimal loss sensitivity and discrete weight removal is fundamentally broken in pretrained LLMs.

What carries the argument

PALS replaces uniform per-layer sparsity with a three-step mechanism: (1) compute the 99th percentile of absolute activations for each layer from a small calibration set, (2) standardize these scores and use them to shift each layer's sparsity target via a linear formula with a ±5% clip, (3) apply Wanda's existing weight scoring within each layer's adjusted budget. The entire procedure adds negligible cost because Wanda already collects the activations.

Load-bearing premise

The specific hyperparameter choices (α=0.05, ±5% bounds, 99th percentile) are validated on the same LLaMA-2-7B results used to evaluate the method, and the alternatives that appear better in single-run ablations are dismissed as within run-to-run noise. The central LLaMA-2 result is well-powered, but the specific configuration that produces it is not independently validated, and the method shows no benefit on Mistral-7B and marginal benefit on LLaMA-3-8B.

What would settle it

If PALS with the same hyperparameters fails to improve over uniform Wanda on models beyond LLaMA-2-7B — as it already does on Mistral-7B — the method may be capturing a property specific to LLaMA-2's layer heterogeneity rather than a general principle of layer-adaptive pruning.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If activation outliers are the right signal for layer importance in pruning, then models designed to have more uniform outlier distributions across layers (like Mistral) may be inherently harder to prune non-uniformly — a property that could become a design criterion for future architectures.
  • The failure of gradient-based allocation challenges the widespread assumption that gradient sensitivity is a good proxy for pruning damage, at least for large discrete weight removal in pretrained models.
  • The ±5% bound being critical suggests that information bottlenecks in even a single layer can cascade through the entire network, which has implications for any compression method that varies capacity across layers.
  • The U-shaped sparsity pattern (less pruning in early and late layers, more in middle layers) aligns with functional accounts of transformer layer roles and could inform other layer-adaptive interventions like quantization or distillation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the benefit of PALS depends on layer heterogeneity, then larger or older models with more uneven layer utilization should benefit more, while well-trained or architecturally uniform models should not — this is testable by measuring activation percentile variance across layers as a predictor of PALS benefit.
  • The clustering of results at α≥0.07 (where bounds clip most layers to 45% or 55%) suggests that a simpler binary allocation scheme might work nearly as well, which would be cheaper to implement and easier to analyze.
  • The gradient failure could extend beyond sparsity allocation to other compression decisions (quantization bit allocation, rank selection in low-rank approximation) where gradient sensitivity is used as a proxy for the impact of discrete capacity reduction.
  • The connection between activation outliers and layer importance predicts that models where outlier dimensions are concentrated in specific layers will benefit more from non-uniform pruning than models where outliers are spread evenly.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes PALS (Percentile-Aware Layerwise Sparsity), a method that allocates per-layer sparsity ratios in one-shot LLM pruning based on the 99th percentile of activation magnitudes, bounded to ±5% around the target ratio. The method plugs into Wanda's existing pipeline with negligible overhead. On LLaMA-2-7B at 50% sparsity, PALS reduces WikiText-2 perplexity from 12.92 to 10.96 (mean over 9 runs, Welch's t(14)=8.1, p<0.001). The benefit is architecture-dependent: marginal on LLaMA-3-8B and absent on Mistral-7B. The paper also reports that gradient-based allocation produces results worse than random, a notable negative finding.

Significance. The paper addresses a clear gap in one-shot LLM pruning: existing methods (Wanda, SparseGPT) apply uniform per-layer sparsity, and the question of whether non-uniform allocation helps is natural and practically relevant. The primary LLaMA-2-7B result is well-powered (9 runs, proper Welch's t-test, calibration set separated from evaluation set). The gradient-failure finding (Table 8) is a genuinely interesting negative result that challenges a common assumption. The authors are commendably transparent about limitations, including the architecture-dependent gains and the single-run ablation tables. The method is simple, practical, and falsifiable.

major comments (3)
  1. §6.2, Table 5: All hyperparameter ablations (α, percentile, bound width) use single runs on a fixed calibration draw, while the primary result uses 9 runs. This is load-bearing because the specific choices of α=0.05, 99th percentile, and ±5% bounds are validated only by these single-run tables. Table 5 illustrates the problem: α=0.07 gives 10.12 PPL (single run) versus α=0.05's 10.96 (9-run mean), and α=0.01 gives 14.23 (worse than the 12.92 baseline). The authors attribute these to seed noise but cannot confirm without multi-run data. At minimum, the key settings bracketing the chosen α (e.g., α=0.03, 0.05, 0.07) should be re-run with multiple seeds to verify that the chosen operating point is not an artifact of the single calibration draw.
  2. §6.6: The 9-run variance estimate (std=0.59) captures only tie-breaking noise with a fixed 128-sample calibration draw from C4. Calibration variance is not measured. The authors acknowledge this gap ('a proper calibration bootstrap is worth doing in future work') but it is central to the method's robustness claim. Since PALS's allocation depends entirely on per-layer activation percentiles from this calibration set, a sensitivity check across at least 2–3 different C4 draws would substantially strengthen the claim that the improvement is not calibration-specific. This is feasible within the paper's experimental scope.
  3. §5.2, Table 2: PALS shows zero benefit on Mistral-7B (6.31 vs. 6.31) and only 0.3% on LLaMA-3-8B. The authors attribute this to architectural differences (§7.2), but an equally plausible explanation is that the hyperparameters are tuned to LLaMA-2-7B's activation distribution under the specific calibration set. Without either (a) multi-run evaluation on the other two models or (b) calibration sensitivity analysis on LLaMA-2, the paper cannot distinguish between 'architecture-specific benefit' and 'calibration-specific overfitting.' This matters for the paper's central practical claim that PALS is a drop-in improvement to Wanda.
minor comments (5)
  1. Figures 1 and 2 use the label 'HALO-v2' instead of 'PALS,' which appears to be a leftover from a previous version of the manuscript.
  2. Table 7: The text states 'the 99th percentile (10.96) is not the best in this table,' but Table 7 does not include the 9-run mean for the 99th percentile—it lists 10.96, which is the 9-run value from Table 1. The other entries are single runs. The comparison is therefore between quantities with different variance estimates, which should be noted.
  3. §6.5, Table 8: The gradient-failure result is striking but also based on single runs. A brief note confirming that the qualitative ordering (gradient worse than random worse than activation) is stable across at least 2 seeds would strengthen this finding.
  4. §3.2, Eq. (3): The clip bounds in Eq. (4) are written as absolute values (s_target ± 0.05), but α=0.07 in Table 5 is labeled '(clipped).' It would help to clarify how often clipping occurs at the chosen α=0.05 setting, perhaps by reporting the fraction of layers at the bounds.
  5. The abstract states 'PALS achieves 10.96 WikiText-2 perplexity versus 12.92 for uniform Wanda' without noting that this is specific to LLaMA-2-7B. Adding 'on LLaMA-2-7B' would prevent misreading.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee correctly identifies that our single-run ablation tables and lack of calibration sensitivity analysis are the main weaknesses of the manuscript. We agree with all three major comments and will address them in revision.

read point-by-point responses

  1. Referee: §6.2, Table 5: All hyperparameter ablations use single runs on a fixed calibration draw, while the primary result uses 9 runs. The specific choices of α=0.05, 99th percentile, and ±5% bounds are validated only by these single-run tables. α=0.07 gives 10.12 PPL (single run) versus α=0.05's 10.96 (9-run mean), and α=0.01 gives 14.23 (worse than the 12.92 baseline). At minimum, the key settings bracketing the chosen α (e.g., α=0.03, 0.05, 0.07) should be re-run with multiple seeds.

    Authors: The referee is correct. The single-run ablation tables are insufficient to justify the chosen operating point, and the non-monotonicity at α=0.01 (14.23, worse than uniform) is exactly the kind of anomaly that requires multi-run verification. We will re-run α=0.01, 0.03, 0.05, and 0.07 with at least 5 seeds each (matching the primary result's methodology) and update Table 5 with means and standard deviations. If the non-monotonicity at α=0.01 persists under multi-run evaluation, we will discuss it as a genuine finding; if it washes out, we will note it was seed noise as we originally suspected. We will also apply the same multi-run treatment to the percentile ablation (Table 7) and bound-width ablation (Table 6) for the settings immediately adjacent to our chosen configuration. revision: yes

  2. Referee: §6.6: The 9-run variance estimate (std=0.59) captures only tie-breaking noise with a fixed 128-sample calibration draw from C4. Calibration variance is not measured. Since PALS's allocation depends entirely on per-layer activation percentiles from this calibration set, a sensitivity check across at least 2–3 different C4 draws would substantially strengthen the claim that the improvement is not calibration-specific.

    Authors: We agree that calibration sensitivity is central to the robustness claim and that our current indirect argument (order-statistic convergence over ~262K tokens) is not a substitute for empirical measurement. We will run PALS on LLaMA-2-7B with 3 independent C4 calibration draws (each 128 samples of 2048 tokens), with multiple seeds per draw, and report the resulting perplexity distribution. This will allow us to decompose variance into tie-breaking noise and calibration noise. We will update §6.6 with these results and revise the discussion accordingly. If calibration variance is non-trivial relative to the improvement margin, we will state this explicitly and discuss implications for practical deployment. revision: yes

  3. Referee: §5.2, Table 2: PALS shows zero benefit on Mistral-7B (6.31 vs. 6.31) and only 0.3% on LLaMA-3-8B. Without either (a) multi-run evaluation on the other two models or (b) calibration sensitivity analysis on LLaMA-2, the paper cannot distinguish between 'architecture-specific benefit' and 'calibration-specific overfitting.' This matters for the paper's central practical claim that PALS is a drop-in improvement to Wanda.

    Authors: This is a fair concern. The combination of single-run evaluation on LLaMA-3 and Mistral, single calibration draw, and single-run ablations means we cannot currently rule out calibration overfitting as an alternative explanation for the architecture-dependent results. We will address this on two fronts. First, the calibration sensitivity analysis on LLaMA-2 (from our response to the second comment) will directly test whether the LLaMA-2 improvement is calibration-specific. Second, we will run multi-seed evaluation (at least 5 seeds) on LLaMA-3-8B and Mistral-7B to determine whether the near-zero and exactly-zero improvements are stable or whether they reflect noise. We will also run at least one alternative calibration draw on LLaMA-3 and Mistral to check whether PALS produces different allocations. If the zero-gain result on Mistral is stable across seeds and calibration draws, the architecture-specific explanation is strengthened; if it shifts, we will revise our claims accordingly. We will also temper the 'drop-in improvement' framing in the abstract and conclusion to make clear that PALS should be validated per-model, consistent with the architecture-dependent results we already report. revision: yes

Circularity Check

0 steps flagged

No circularity found — PALS's derivation is self-contained and its inputs (activation percentiles from calibration data) are not defined in terms of its outputs (per-layer sparsity or perplexity).

full rationale

The paper's derivation chain is straightforward and non-circular. PALS computes the 99th percentile of activation magnitudes from calibration data (Eq. 1), standardizes them (Eq. 2), maps them to bounded per-layer sparsity ratios (Eqs. 3–4), and then applies Wanda's existing weight scoring (Eq. 5). At no point is an output quantity (perplexity, sparsity ratio) used to define an input quantity (activation percentile, importance score). The activation statistics are computed from a fixed C4 calibration draw, not fitted to the target perplexity. The hyperparameters (α=0.05, 99th percentile, ±5% bounds) are selected from ablation tables (Tables 5–7), but the authors are transparent that these are single-run ablations, that α=0.07 and the 99.5th percentile give better single-run numbers, and that the differences are within run-to-run variance. This is a generalization/statistical-power concern, not circularity: the ablation inputs (different α values, different percentiles, different bounds) are genuinely different from the outputs (perplexity), and the authors do not claim the hyperparameter choices are 'predictions' or 'first-principles results.' The paper's premises rest on external citations (Dettmers et al. 2022 for activation outliers, Sun et al. 2024 for Wanda scoring) that are not authored by the present authors. No uniqueness theorem, no self-citation chain, no ansatz smuggled via citation, and no renaming of a known result. The skeptic's concerns about calibration-set dependence and single-run ablations are valid correctness/generalization risks but do not constitute circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

PALS introduces no new entities, particles, or forces. It reuses Wanda's scoring and activation collection. The three free parameters (α, percentile, bound width) are all chosen from ablations on the same model used for the primary result, though the authors are transparent about this. The core axioms are domain assumptions inherited from prior work (Wanda, Dettmers et al.) rather than ad hoc postulates.

free parameters (3)
  • α (adaptation strength) = 0.05
    Chosen from ablation (Table 5); controls how much standardized importance shifts per-layer sparsity. Authors note α=0.07 gives better single-run PPL but chose 0.05 as a transition point.
  • Percentile for activation importance = 99
    Chosen from ablation (Table 7); 95th and 99.5th percentiles give better single-run PPL. Authors state they chose 99th before running the ablation based on connection to Dettmers et al. 2022.
  • Sparsity bound width = ±5%
    Chosen from ablation (Table 6); ±5% gives best result. Wider bounds degrade sharply. Whether ±5% is universally best is acknowledged as an open question.
axioms (3)
  • domain assumption Activation outlier magnitude (99th percentile) is a valid proxy for layer importance in pruning
    §3.1 and §7.1: the paper connects this to Dettmers et al. 2022's finding that activation outliers are critical for model quality, but does not prove the connection; it is a hypothesis supported empirically by the results.
  • domain assumption Wanda's weight scoring (|w|·mean(|a|)) is a sound basis for within-layer pruning
    §3.3: PALS uses Wanda's scoring unchanged; this is inherited from Sun et al. 2024 and is not re-validated here.
  • domain assumption Calibration activations from 128 C4 samples are sufficient for stable percentile estimates
    §6.6: the authors argue order statistics converge quickly with ~262K tokens but acknowledge they did not run a formal calibration sensitivity experiment.

pith-pipeline@v1.1.0-glm · 13218 in / 3001 out tokens · 552839 ms · 2026-07-09T06:59:32.757309+00:00 · methodology

0 comments
read the original abstract

One-shot pruning methods like Wanda and SparseGPT apply the same sparsity ratio to every layer of a transformer, ignoring known variation in layer importance. We propose PALS (Percentile-Aware Layerwise Sparsity), which adjusts per-layer sparsity based on the 99th percentile of activation magnitudes, bounded to $\pm 5\%$ around the target ratio. On LLaMA-2-7B at 50\% sparsity, PALS achieves 10.96 WikiText-2 perplexity versus 12.92 for uniform Wanda (mean over 9 runs, $p < 0.001$). The benefit is architecture-dependent: LLaMA-3-8B shows marginal gains and Mistral-7B shows none. We also find that gradient-based allocation -- the seemingly more principled approach -- produces results worse than random, suggesting that gradient magnitude does not predict the impact of discrete weight removal. PALS adds negligible cost to the pruning pipeline and requires no fine-tuning.

Figures

Figures reproduced from arXiv: 2607.07557 by Alexey Shvets, Yazdan Jamshidi.

Figure 1
Figure 1. Figure 1: WikiText-2 perplexity at 50% sparsity on LLaMA-2-7B. Error bars: [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: WikiText-2 perplexity across sparsity levels on LLaMA-2-7B. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-layer sparsity assigned by PALS at 50% target. Layer 0 is closest to the input embed [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 27 canonical work pages · 12 internal anchors

  1. [1]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

    work page

  2. [2]

    PaLM: Scaling Language Modeling with Pathways

    Palm: Scaling language modeling with pathways , author=. arXiv preprint arXiv:2204.02311 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    LLaMA: Open and Efficient Foundation Language Models

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Journal of Machine Learning Research , volume=

    Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of Machine Learning Research , volume=

    work page

  5. [5]

    Advances in neural information processing systems , pages=

    Optimal brain damage , author=. Advances in neural information processing systems , pages=

    work page

  6. [6]

    Advances in neural information processing systems , pages=

    Learning both weights and connections for efficient neural network , author=. Advances in neural information processing systems , pages=

    work page

  7. [7]

    SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

    SparseGPT: Massive language models can be accurately pruned in one-shot , author=. arXiv preprint arXiv:2301.00774 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    A Simple and Effective Pruning Approach for Large Language Models

    A simple and effective pruning approach for large language models , author=. arXiv preprint arXiv:2306.11695 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

    Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned , author=. arXiv preprint arXiv:1905.09418 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1905

  10. [10]

    Advances in neural information processing systems , volume=

    Are sixteen heads really better than one? , author=. Advances in neural information processing systems , volume=

    work page

  11. [11]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    Structured pruning of large language models , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    work page 2020

  12. [12]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    NISP: Pruning networks using neuron importance score propagation , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    work page

  13. [13]

    International Conference on Learning Representations , year=

    Rethinking the value of network pruning , author=. International Conference on Learning Representations , year=

    work page

  14. [14]

    Advances in Neural Information Processing Systems , volume=

    Optimal Brain Compression: A framework for accurate post-training quantization and pruning , author=. Advances in Neural Information Processing Systems , volume=

    work page

  15. [15]

    Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

    Sheared llama: Accelerating language model pre-training via structured pruning , author=. arXiv preprint arXiv:2310.06694 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Advances in Neural Information Processing Systems , volume=

    LLM-Pruner: On the structural pruning of large language models , author=. Advances in Neural Information Processing Systems , volume=

    work page

  17. [17]

    SliceGPT: Compress Large Language Models by Deleting Rows and Columns

    SliceGPT: Compress large language models by deleting rows and columns , author=. arXiv preprint arXiv:2401.15024 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    International Conference on Learning Representations , year=

    The lottery ticket hypothesis: Finding sparse, trainable neural networks , author=. International Conference on Learning Representations , year=

    work page

  19. [19]

    Faster gaze prediction with dense networks and Fisher pruning

    Faster gaze prediction with dense networks and Fisher pruning , author=. arXiv preprint arXiv:1801.05787 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning

    Pruning meets low-rank parameter-efficient fine-tuning , author=. arXiv preprint arXiv:2305.18403 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    Pointer Sentinel Mixture Models

    Pointer sentinel mixture models , author=. arXiv preprint arXiv:1609.07843 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    PIQA: Reasoning about physical commonsense in natural language , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    work page

  23. [23]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=

    HellaSwag: Can a machine really finish your sentence? , author=. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=

    work page

  24. [24]

    BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

    BoolQ: Exploring the surprising difficulty of natural yes/no questions , author=. arXiv preprint arXiv:1905.10044 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1905

  25. [25]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Think you have solved question answering? Try ARC, the AI2 reasoning challenge , author=. arXiv preprint arXiv:1803.05457 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    Advances in Neural Information Processing Systems , volume=

    GPT3.int8(): 8-bit matrix multiplication for transformers at scale , author=. Advances in Neural Information Processing Systems , volume=

    work page

  27. [27]

    A framework for few-shot language model evaluation , author=

    work page