Pith. sign in

REVIEW 3 major objections 5 minor 32 references

Post-Training Statistical Calibration for Higher Activation Sparsity

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SCAP prunes FC inputs and centers activation modes, hitting 48.5% FFN sparsity on Mistral-7B at iso quality versus CATS' 33.3% and a 1.5x relative decoding speedup.

desk verdict SCAP is a practical extension of CATS with a genuinely useful mode-centering trick; the 1.5x speedup claim is plausible but rests on borrowed CATS numbers and needs a head-to-head rerun. read the letter →

arxiv 2412.07174 v1 pith:TOAWZZWE submitted 2024-12-10 cs.LG cs.AI

Vui Seng Chua , Yujie Pan , Nilesh Jain This is my paper · ORCID
classification cs.LGcs.AI
keywords activationsparsitypost-trainingpruningfeed-forwardnetworksmode-centeringcalibrationL1thresholdingLLMdecodingspeedupSiLUandGELUactivationsTransformerinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SCAP is a post-training method for making large language models sparser without retraining or uptraining. The paper argues that pruning the input activations of fully-connected layers, rather than the outputs of activation functions, is a more general and flexible way to induce sparsity, and that a simple Mode-Centering step makes this much more effective on GELU and SiLU models. The headline result is that at equal zero-shot task accuracy on Mistral-7B, SCAP reaches 48.5% feed-forward sparsity versus CATS' 33.3%, translating into a 27.1% decoding speedup versus 17.7% for CATS—a 1.5x relative gain. The paper also shows the method extends to MoE, Mamba2, vision transformers, and pre-quantized models, which matters because it promises practical, low-cost activation sparsity for modern non-ReLU LLMs.

What carries the argument

The central machinery is threshold-based activation pruning on FC inputs (Eq. 2): each element $X_{ij}$ is kept only if $|X_{ij}| > \tau$, with $\tau$ calibrated as a quantile of saved calibration activations. The second ingredient is Mode-Centering calibration (Eq. 6): a static mode $\eta$ is estimated offline and subtracted from the activation, with the compensation folded into the fused bias, so the distribution's peak moves to zero and many more values fall below the pruning threshold. Together they decouple sparsity across Up, Gate, and Down projections, which the paper identifies as the source of its Pareto advantage over CATS.

What would settle it

Compare pruned and unpruned outputs layer by layer on held-out text: if many activations just below the calibrated threshold fall in weight columns with large column norms, the reconstruction error of the sparse layer will grow faster than the reported task-accuracy curves; that is directly measurable.

Watch

Extended reading notes

Core claim

SCAP applies an L1-magnitude pruner to the input activation $X$ of any fully-connected layer, masking elements with $|X_{ij}| < \tau$, where $\tau$ is set by a quantile of calibration activations. Because the pruner sits at the FC input, the same kernel and calibration procedure apply to the Up, Gate, and Down projections of GLU FFNs, to attention projections, and to non-GLU GELU networks. For activations whose distribution peaks away from zero, the paper introduces Mode-Centering: it estimates the mode $\eta$ offline, computes $Y = (X - \eta)W + b_{\mathrm{fused}}$, and folds the compensating $\eta W$ into the bias, shifting the density peak to zero and making L1 thresholding far more effective. The paper's central empirical claim is that at equal zero-shot accuracy on Mistral-7B, SCAP achieves 48.5% FFN sparsity versus CATS' 33.3%, producing a 27.1% decoding speedup versus 17.7%—a 1.5x relative improvement.

Load-bearing premise

A small activation magnitude is treated as a reliable sign that the corresponding weight column contributes little to the output, even though the paper does not examine weight magnitudes or error propagation.

Editorial extensions

If this is right

  • Post-training activation sparsity no longer depends on ReLU activations; SCAP works on SiLU/GELU models like Mistral, Llama-2, Falcon, and MPT without any uptraining.
  • Because sparsity is applied at FC inputs, SCAP can set different sparsity levels for Up, Gate, and Down projections, which is what yields higher FFN sparsity at a given task-accuracy tolerance than CATS.
  • Mode-Centering raises Down-projection prunability in non-GLU FFNs: Falcon-7B goes from 30.5% to 50.3% sparsity and MPT-7B from 12.7% to 57.4% within about 1% relative task drop.
  • The method transfers across model families—Mixtral, Mamba2, DeiT vision transformers, and pre-quantized Llama models—all at roughly -1% relative task accuracy.
  • Because SCAP requires only a single GPU for calibration and no predictor training, it makes activation sparsification feasible on modest hardware in a post-training workflow.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same mode-centering logic could be pushed further with per-layer affine calibration beyond a scalar shift—such as a learned shift plus scale—which might concentrate even more activation mass near zero, though the paper only tests a single scalar mode.
  • SCAP's speedup numbers assume batch-1 decoding; the paper's own appendix shows overlapping sparsity drops sharply as beam width or batch size grows, so the practical 1.5x advantage is likely smaller in high-throughput batched serving.
  • Pruning at FC inputs changes the activation distribution seen by every downstream layer, and the paper validates this only through aggregated task accuracy; a direct layer-by-layer error-propagation study would be a natural next test.
  • The paper compares against trained sparsity methods like TurboSparse at lower sparsity but far lower compute cost; a natural extension is combining SCAP with parameter-efficient fine-tuning to close the sparsity gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SCAP, a post-training activation pruning method that induces sparsity in the input activations of fully-connected (FC) layers of Transformers by thresholding on the L1 magnitude of activations, with a Mode-Centering pre-calibration step for activation distributions whose mode is not near zero. The method is calibration-based, requiring no retraining or predictor networks. The authors claim that SCAP achieves a better Pareto trade-off between FFN sparsity and zero-shot task accuracy than the prior post-training method CATS, and they report a 1.5x relative decoding speedup over CATS at iso model quality on Mistral-7B. The paper additionally demonstrates SCAP on a wide range of models, including non-GLU decoders, MoE, Mamba2, vision encoders, and pre-quantized models.

Significance. If the central claims are substantiated, SCAP is a practically useful contribution: it offers a generic, calibration-only sparsification approach that applies to input activations of any FC layer, and the Mode-Centering technique provides a simple fix for non-zero-centered activation distributions. The paper is commendable for releasing code, for covering a broad set of model families, and for including a kernel-level discussion of decoding acceleration. The main significance is conditional on the reliability of the headline iso-quality comparison, which is currently not fully supported by the evidence presented.

major comments (3)
  1. [Section 4.1 / D.1 / Table 2] The central claim of a 1.5x decoding speedup at iso quality rests on a single pair of configurations (Table 2) in which the CATS accuracy (74.2%) is taken directly from the CATS paper rather than measured with the same evaluation harness, as stated in Section D.1. Per-task differences between CATS and SCAP in Table 5 reach about 1.7 points (e.g., Arc-C 50.4 vs 48.7), and no variance, seeds, or repeated evaluations are reported, so the equality at one decimal place does not establish that the two points are iso-quality. The paper should rerun CATS under the same harness (or provide paired bootstrap confidence intervals) and also verify that the internally reproduced CATS kernel speedup (17.7%) is consistent with the official implementation, since a slower CATS kernel would inflate the ratio. Without this, the headline 1.5x relative speedup is not supported.
  2. [Appendix B, Algorithm 2] Algorithm 2 is inconsistent with the method description. In a SwiGLU FFN, the input to the Down projection is the gated activation (product of the Up path and the SiLU of the Gate path), but line 8 of Algorithm 2 passes the original input x to SCAP_FC for the Down projection (y ← SCAP_FC(τgated, x, Wdown)), rather than the computed zgated. As written, the algorithm computes the wrong operation and cannot reproduce the Down sparsities reported in Tables 5 and 6. The pseudocode should be corrected and the variable definitions clarified.
  3. [Eq. (2)] The importance measure in Eq. (2) is the L1 magnitude of the input activation, but the paper does not analyze when this is a valid proxy for the actual contribution of that activation to the output, given that weight column norms vary. This matters for the claimed generality of SCAP to all FC layers (attention QKV/output projections, non-GLU Down projections, and MoE layers), where the assumption has not been previously validated. The paper should provide per-layer evidence, such as output reconstruction error or a comparison with a weight-aware importance criterion, to show that L1 thresholding is safe for the new layer types; the downstream task accuracy alone is an indirect check that mixes in the robustness of the task metric.
minor comments (5)
  1. [Appendix B, Algorithm 2] Algorithm 2, line 3 uses the symbol 'xηW' which is not defined; either use xW for the non-mode-centered case or explicitly define the mode-centered input xη in the pseudocode, and align the threshold comparison (≥ in Alg. 2 vs > in Eq. 2).
  2. [Section 4.2] The decoding speedups in Table 2 are reported to one decimal place without error bars or the number of repeated runs; please state the measurement uncertainty or provide the number of independent repetitions.
  3. [Table 4] The 'Relative (%)' column should specify that it is the relative change in the task metric (zero-shot average or ImageNet top-1) against the unpruned baseline, and the exact task set used for each model should be listed for the non-LLM entries.
  4. [Figure 3] The subcaptions in Figure 3 are too terse: identify the model and exact tensor for each panel (e.g., 'post-LayerNorm input to Up/Gate in Mistral-7B') so that the distributions can be reproduced.
  5. [Section 4.4] The statement that SCAP 'outperformed TurboSparse' on the remaining tasks is not supported by a per-task table; adding the per-task scores would make the comparison transparent and would also clarify the large GSM8K gap.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SCAP's sparsity targets are calibration inputs, and its accuracy and speedup claims are measured out-of-sample; the borrowed CATS baseline is a fairness caveat, not a circular derivation.

full rationale

SCAP's pipeline is empirical rather than derivational: pruning thresholds are calibrated to a C4 sample as quantiles of |X|, mode shifts are estimated from the same sample, and all headline quantities—downstream zero-shot accuracy, observed sparsity on evaluation tasks, and decoding latency—are measured on held-out data or benchmarked kernels. Equation (2) defines tau = Quantile(|X_calib|, s), so the target sparsity s is an explicit, user-chosen input, not a quantity that the paper derives or 'predicts' from the method; the paper consistently treats s as a sweep parameter and reports the actual activation sparsity observed during task evaluation, with Appendix A checking target-versus-actual agreement. Mode-centering (Eqs. 4-6) is algebraically exact under a static shift, and its benefit is judged by downstream task accuracy after pruning, not by the calibration fit itself. There are no load-bearing self-citations: all prior-work citations, including CATS, are external, and the comparison to CATS is an empirical benchmark rather than a result forced by the paper's own definitions. The only notable concern is comparison fairness: Appendix D.1 states 'We directly referenced the results reported in [12]' for CATS Pareto points, and Table 2's iso-quality pair uses CATS's reported 74.2% accuracy rather than a fresh CATS evaluation in the same harness. That is a validity and robustness caveat about external benchmark borrowing, not a circular reduction of SCAP's claims to its inputs; no fitted parameter is renamed as a prediction, and the 48.5% FFN sparsity is an observed test-time quantity tied to a chosen (40%, 60%) calibration target.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

SCAP introduces no new physical or architectural entity. Its load-bearing degrees of freedom are per-layer mode shifts and per-group pruning thresholds, both estimated from calibration data, plus the grid-selected sparsity targets. The main unstated premise is that L1 magnitude on input activations is a valid importance measure, which is an empirical assumption rather than a derived result.

free parameters (3)
  • per-layer mode shift eta = median or KDE mode of calibration activations
    Estimated offline from C4 calibration slices; used to shift activation distributions before L1 pruning (Section 3). One scalar per targeted activation distribution.
  • pruning threshold tau per activation group = quantile of calibration |X| at chosen sparsity level, e.g., 40% for up/gate and 60% for down
    Determined by quantile calibration on 64 C4 slices; group-level targets swept in 10% increments (Section 4).
  • target group sparsities s_up/gate and s_down = e.g., 40% and 60% for the iso-quality Mistral-7B point
    Chosen by grid search over 20-80% and filtered by a -1% task accuracy tolerance; this is model selection on evaluation tasks (Section 4).
assumptions (3)
  • domain assumption L1 magnitude of an input activation is a sufficient importance signal for pruning
    Eq. 2 defines the pruner as thresholding on |Xij| without consulting weight magnitudes; the paper relies on this for quality retention across all models.
  • domain assumption Static mode estimated offline remains valid for deployment inputs
    Section 3 assumes eta can be fixed during pre-deployment and fused into bias; distribution shift between calibration and deployment is not quantified.
  • domain assumption Calibration sample of 64 C4 slices is representative of downstream task activations
    Section 4 uses 64 slices of 256 tokens to set all thresholds and modes; Appendix A checks sparsity alignment on tasks but not worst-case shift.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Post-Training Statistical Calibration for Higher Activation Sparsity." pith.science (2026). https://pith.science/paper/TOAWZZWE

@misc{pith2026241207174,
  author       = {Pith},
  title        = {Pith review of: Post-Training Statistical Calibration for Higher Activation Sparsity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TOAWZZWE}},
  note         = {Machine review of arXiv:2412.07174}
}
read the original abstract

We present Statistical Calibrated Activation Pruning (SCAP), a post-training activation pruning framework that (1) generalizes sparsification by input activations of Fully-Connected layers for generic and flexible application across Transformers, and (2) features a simple Mode-Centering technique to pre-calibrate activation distributions for maximizing post-training sparsity. Our results demonstrate robust Pareto efficiency compared to prior methods, translating to a 1.5x additional LLM decoding speedup against CATS at iso model quality. SCAP effectiveness is empirically verified across a wide range of models, including recent Transformer Decoders, MoE, Mamba2, Encoding Transformer, and pre-quantized models, highlighting its practicality and scalability. The code is available at: https://github.com/IntelLabs/SCAP.

Figures

Figures reproduced from arXiv: 2412.07174 by the authors.

Figure 1
Figure 1. ReLU output is sparse while SiLU and GELU outputs are dense. Extracted from [17]. In contrast, post-training optimization offers a simpler and more cost-effective approach, as ex￾emplified by weight quantization methods such as GPTQ[8] and AWQ[14]. These techniques determine quantization parameters through calibration with a small set of text prompts on pretrained or instruction-tuned LLMs, making them highly effici… view at source ↗
Figure 2
Figure 2. Activation Sparsification across methods on SwiGLU [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Activation distributions for (a) post-normalization inputs to [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Effect of Mode-Centering Calibration on Activation Sparsity [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Computational graph of an FC layer with mode-centered and pruned input activation [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Pareto front of CATS and SCAP (Ours) across LMs, with numbers provided in Section D.1 [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Activation Pruning with and without Mode Centering on input of Down Projection (GELU [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Target vs. Actual input Sparsity of F Cdown across various Transformer Layers in Llama. Each point represents for a different task, with its empirical sparsity closely aligning with diagonal line, indicating a strong correlation with the target. This illustrates the ef…
Figure 9
Figure 9. Figure 9: SwiGLU Latency Scaling w.r.t FFN Sparsity across implementations C Acceleration Challenges of Batched Sparse Activation Inference acceleration through activation sparsity has primarily focused on the token-to-token decod￾ing phase of language models, operating under th…
Figure 10
Figure 10. Figure 10: Structured Sparsity declines with beam width size (a proxy for high batch input) [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 9 canonical work pages

  1. [1]

    AbouElhamayed, Jordan Dotzel, Zhiru Zhang, Alexander M

    Yash Akhauri, Ahmed F. AbouElhamayed, Jordan Dotzel, Zhiru Zhang, Alexander M. Rush, Safeen Huda, and Mohamed S. Abdelfattah. ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models, June 2024. arXiv:2406.16635 [cs]

    arXiv 2024

  2. [2]

    Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar

    Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C. Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. LLM in a flash: Efficient Large Language Model Inference with Limited Memory, July 2024. arXiv:2312.11514 [cs]

    arXiv 2024

  3. [3]

    The Falcon Series of Open Language Models, November 2023

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malar- tic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The Falcon Series of Open Language Models, November 2023. arXiv:2311.16867 [cs]

    arXiv 2023

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    2021

  5. [5]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

    arXiv 2022

  6. [6]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009

    2009

  7. [7]

    Lighteval: A lightweight framework for llm evaluation, 2023

    Clémentine Fourrier, Nathan Habib, Thomas Wolf, and Lewis Tunstall. Lighteval: A lightweight framework for llm evaluation, 2023

    2023

  8. [8]

    GPTQ: Accurate Post- Training Quantization for Generative Pre-trained Transformers, March 2023

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate Post- Training Quantization for Generative Pre-trained Transformers, March 2023. arXiv:2210.17323 [cs]. 9

    arXiv 2023

Show all 32 references
  1. [9]

    A framework for few-shot language model evaluation, 12 2023

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2023

  2. [10]

    Learning both Weights and Connections for Efficient Neural Network

    Song Han, Jeff Pool, John Tran, and William Dally. Learning both Weights and Connections for Efficient Neural Network. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015

    2015

  3. [11]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Syste...

    2023

  4. [12]

    CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models, April 2024

    Je-Yong Lee, Donghyun Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini. CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models, April 2024. arXiv:2404.08763 [cs]

    2024 arXiv

  5. [13]

    The Lazy Neuron Phenomenon: On Emergence Of Activation Sparsity In Transformers

    Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, and Sanjiv Kumar. The Lazy Neuron Phenomenon: On Emergence Of Activation Sparsity In Transformers. 2023

    2023

  6. [14]

    AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, July 2024

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, July 2024. arXiv:2306.00978 [cs]

    2024 arXiv

  7. [15]

    Deja Vu: contextual sparsity for efficient LLMs at inference time

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas- tava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. Deja Vu: contextual sparsity for efficient LLMs at inference time. In Proceedings of the 40th International Conference on Mac...

  8. [16]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

    2016

  9. [17]

    del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar

    Seyed Iman Mirzadeh, Keivan Alizadeh-Vahid, Sachin Mehta, Carlo C. del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models. October 2023

    2023

  10. [18]

    Orca-math: Unlocking the potential of slms in grade school math, 2024

    Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math, 2024

    2024

  11. [19]

    Openwebmath: An open dataset of high-quality mathematical web text, 2023

    Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023

    2023

  12. [20]

    Efficiently Scaling Transformer Inference

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently Scaling Transformer Inference. Proceedings of Machine Learning and Systems, 5:606–624, March 2023

    2023

  13. [21]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019

    2019

  14. [22]

    Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs | Databricks Blog, May 2023

    Mosaic AI Research. Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs | Databricks Blog, May 2023

    2023

  15. [23]

    GLU Variants Improve Transformer, February 2020

    Noam Shazeer. GLU Variants Improve Transformer, February 2020. arXiv:2002.05202 [cs, stat]

    2020 arXiv

  16. [24]

    ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models, July 2024

    Chenyang Song, Xu Han, Zhengyan Zhang, Shengding Hu, Xiyu Shi, Kuai Li, Chen Chen, Zhiyuan Liu, Guangli Li, Tao Yang, and Maosong Sun. ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models, July 2024. arXiv:2402.13516 [cs]. 10

    2024 arXiv

  17. [25]

    PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU, December 2023

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU, December 2023. arXiv:2312.12456 [cs]

    2023 arXiv

  18. [26]

    Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters, June 2024

    Yixin Song, Haotong Xie, Zhengyan Zhang, Bo Wen, Li Ma, Zeyu Mi, and Haibo Chen. Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters, June 2024. arXiv:2406.05955 [cs]

    2024 arXiv

  19. [27]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A Simple and Effective Pruning Approach for Large Language Models, May 2024. arXiv:2306.11695 [cs]

    2024 arXiv

  20. [28]

    Deit iii: Revenge of the vit

    Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit iii: Revenge of the vit. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIV, page 516–533, Berlin, Heidelberg, 2022. Springer-Verlag

    2022

  21. [29]

    Gpt-j-6b: A 6 billion parameter autoregressive language model, 2021

    Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model, 2021

    2021

  22. [30]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2019

  23. [31]

    Orca: A distributed serving system for Transformer-Based generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) , pages 521–538, Carlsbad, CA, July 2022...

    2022

  24. [32]

    ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMs, February 2024

    Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, and Maosong Sun. ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMs, February 2024. arXiv:2402.03804 [cs]. 11 A Target vs Actual Activation Spars...

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.