Pith. sign in

REVIEW 4 major objections 4 minor 52 references

SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SparseSSM is a training-free, one-shot pruning framework that removes half of Mamba's state-space weights with no zero-shot accuracy loss.

desk verdict Useful, well-executed pruning recipe for Mamba, but the OBS derivation in Theorem 1 does not survive scrutiny—the final score is just a Wanda-style weight-activation product. read the letter →

arxiv 2506.09613 v1 pith:AM6DNBUL submitted 2025-06-11 cs.LG

Kaiwen Tuo , Huan Wang This is my paper · ORCID
classification cs.LG
keywords statespacemodelsMambaone-shotpruningoptimalbrainsurgeonHessiansaliencyLLMcompressiontraining-freestructuredsparsity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that Mamba-based language models can have half of their state-space weights removed in a single pass, with no fine-tuning and no zero-shot accuracy loss. It is the first work to adapt the optimal brain surgeon (OBS) second-order pruning framework to the selective state-space module, whose transition matrix is shared across time steps and discretized at inference. The paper derives an approximate saliency score for this matrix and a protocol for merging per-time-step masks, plus a sensitivity analysis showing which feed-forward projections tolerate pruning. If correct, this makes state-space LLMs as compressible as Transformers at the same sparsity, reducing deployment cost.

What carries the argument

The load-bearing object is the approximate second-order saliency score of Theorem 1, which reduces the OBS importance of each element of the diagonal transition matrix $A_{\log}$ to a product of its squared value and the accumulated squared hidden-state activation across all time steps, $I^{\log}_{d,n} \propto A_{\log,d,n}^2 \sum_{b,i} h_{b,i-1,d,n}^2$. A three-phase aggregation protocol (Algorithm 1) converts each time step's candidate mask into a global mask by selecting the entries most frequently flagged as least important, handling the time-sharing of $A_{\log}$. For the feed-forward component, a Hessian-trace sensitivity score assigns lower sparsity to the sensitive in_proj and out_proj projections while holding the global sparsity budget. The same scores extend to N:M and fully structured column pruning by summing importance over the state dimension.

What would settle it

On a small Mamba model, compute the exact diagonal Hessian of the SSM loss by double backpropagation and measure the rank correlation against the SparseSSM score on $A_{\log}$; low correlation would mean the OBS derivation is not what carries the results. Alternatively, replace the score by the heuristic $A_{\log,d,n}^2$-weighted activation sum with the same mask voting and check whether perplexity is unchanged.

Watch

Extended reading notes

Core claim

The central claim is that the time-shared, discretized state-transition parameter $A_{\log}$ of a Mamba SSM can be pruned one-shot at 50% sparsity with no zero-shot degradation, provided the importance ranking is derived from the OBS framework tailored to the SSM's recurrence. Theorem 1 shows that, under a diagonal-state assumption and after dropping third-order gradient terms, the OBS saliency for $A_{\log,d,n}$ reduces to $A_{\log,d,n}^2$ times the summed squared hidden activations across batches and time steps. The paper then aggregates per-time-step candidate masks by pruning frequency to respect the parameter's time-sharing, and uses Hessian-trace sensitivity to allocate sparsity across FFN modules. On Mamba-130M through Mamba-1.4B, the pruned models match or beat dense zero-shot accuracy on several benchmarks while outperforming magnitude pruning, SparseGPT, and Mamba-Shedder.

Load-bearing premise

The ranking of which weights to drop is trustworthy only if the dropped backpropagated-gradient terms and the absorbed factor $\delta^2 e^{2\delta A}$ are truly negligible across all weights; if they vary across weights, the score is a heuristic product rather than a second-order measure.

Editorial extensions

If this is right

  • Mamba LLMs up to 1.4B parameters can be pruned to 50% SSM sparsity in one shot with no zero-shot accuracy loss, and with only a small perplexity increase.
  • The method beats magnitude pruning, SparseGPT, and Mamba-Shedder at every sparsity level from 40% to 80% on Mamba-130M, 370M, 790M, and 1.4B.
  • The same saliency score extends to 2:4 and 4:8 semi-structured patterns and to fully structured column pruning, which yields a 1.72x speedup on SSM inference at 50% sparsity.
  • The sensitivity analysis identifies in_proj and out_proj as low-tolerance modules, so sparsity should be concentrated elsewhere.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the final score is a simple product of squared weight and activation energy, the method's gains over magnitude pruning could come mostly from the activation weighting; an ablation that replaces Theorem 1's score by weight-times-activation without the OBS derivation would isolate the contribution.
  • Pruned entries cluster in whole columns of $A_{\log}$, hinting that the state dimension $N$ is overprovisioned in trained Mamba models; a smaller state size trained from scratch might match the pruned model's quality.
  • The pruning-frequency mask voting for time-shared parameters could transfer to other recurrent or linear-attention architectures whose weights are shared across sequence steps, although the paper only demonstrates it on Mamba.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. SparseSSM proposes a one-shot, training-free pruning method for the selective state-space (SSM) modules of Mamba-based language models. The central idea is to adapt the Optimal Brain Surgeon (OBS) framework to the time-shared, discretized diagonal state-transition matrix A_log. The paper derives an approximate second-order saliency score that reduces to the product of the squared weight magnitude and the accumulated squared hidden-state activation (Theorem 1, Eq. (6)), introduces a time-step mask aggregation procedure (Algorithm 1), adds a Hessian-trace-based sensitivity analysis to allocate sparsity across FFN modules (Eq. (7)), and reports extensive experiments on Mamba-130M to 1.4B, including unstructured, N:M, and structured sparsity. The empirical results consistently beat the chosen baselines and show that pruning 50% of SSM weights can be done without catastrophic degradation, although the often-claimed absence of zero-shot accuracy loss is not supported by the reported averages.

Significance. If the theoretical derivation and empirical claims hold, SparseSSM would be a useful contribution: it extends second-order post-training pruning to a non-Transformer architecture that is poorly covered by existing methods, and the proposed time-step aggregation and module-sensitivity analysis are of independent interest. The paper's strengths are its broad evaluation across four model scales, several sparsity levels, and multiple downstream tasks, and the fact that the method is computationally light (pruning times in Table 7 are modest). However, the load-bearing OBS derivation in Appendix A is not a valid derivation of the stated Hessian approximation; the surviving score is essentially a Wanda-style heuristic. Because the claimed theoretical novelty rests on this derivation, the paper cannot be accepted in its current form. The empirical part is promising but needs corrected claims about calibration and zero-shot loss, plus error bars or repeated-run statistics to support the 'no loss' statement.

major comments (4)
  1. [Appendix A, Eqs. (13)-(20); Theorem 1] The proof of Theorem 1 contains unjustified approximations that are load-bearing for the claimed OBS extension. In Eq. (12), differentiating Eq. (11) with respect to A_log produces a term involving ∂h_{b,i-1}/∂A_log, which is nonzero because h_{b,i-1} depends on A_log through the full recurrence; this term is of the same order as the retained terms and cannot be dismissed as third-order. Eq. (17) is not a valid identity: ∂²L/∂(A_d,n h_{b,i-1,d,n})² is not equal to (∂L/∂h_{b,i,d,n}) h_{b,i-1,d,n}. Moreover, Eq. (20) absorbs A² δ² e^{2δA}, which depends on the very parameter A_log being ranked, into a global constant κ; in particular, A = e^{-A_log}, so the exponential factor varies with the parameter. As written, the derivation reduces the OBS saliency to A_log² Σ h², which is the product of squared weight magnitude and accumulated squared activation, i.e., a Wanda-style heuristic, not a demonstrated second-order score. To keep the theoretical claim, the authors need to either provide a rigorous derivation with explicit control of these terms, or recharacterize the score as a heuristic and validate it against actual Hessian-based rankings on small models.
  2. [Abstract, §1 bullet, and §4] The paper states that pruning is performed 'without fine-tuning or calibration' (Abstract and contribution bullet), but Section 4 explicitly says that 128 contiguous segments of 2048 tokens from WikiText-2 are used as calibration data, and Appendix B.1 sets Nsample = 64. The method uses this calibration data to collect hidden states and compute Hessian traces, so the 'without calibration' claim is contradicted by the experimental protocol. Additionally, the claim of 'no zero-shot accuracy loss' is not supported by the reported tables: in Table 1, the average zero-shot accuracy against the dense model drops from 43.36 to 42.35 (Mamba-130M), 50.64 to 49.51 (Mamba-790M), and 53.98 to 51.54 (Mamba-1.4B), and only Mamba-370M is essentially flat (47.49 vs 47.47). Table 2 shows larger drops for whole-model pruning. These overstatements should be corrected, and the tables should report error bars or standard deviations over repeated calibration subsets to substantiate 'no loss' claims.
  3. [Eq. (7) and §3.4; Appendix B.1 Eq. (27)] The sensitivity-aware sparsity assignment formula sparsity = 1 - p - α + (2α id)/(N-1) is monotonically increasing in the sensitivity-rank index id. The text states that 'higher-sensitivity weights (larger id) are assigned lower sparsity,' which is the opposite of what the formula computes. If id is sorted in ascending order of importance, then the formula gives higher sparsity to more sensitive weights, which contradicts the stated design; if id is intended in descending order, the description must be clarified. This is a load-bearing inconsistency in the FFN pruning strategy, and it prevents the reader from interpreting Eq. (7) as claimed. It also raises the question of whether the reported gains over SparseGPT in the FFN pruning experiments are produced by the intended mechanism.
  4. [Algorithm 1 and §5 Conclusion] The conclusion states that the method 'reconstructs remaining weights to minimize output error,' but Algorithm 1 only computes a mask and zeroes out the selected entries of A_log; no weight reconstruction or update step is present. The paper should either explicitly state that no reconstruction is applied (and justify this via the diagonal structure of the Hessian, if that is the case) or implement the OBS update rule. The current presentation conflates the OBS saliency framework with SparseGPT-style weight reconstruction, and the claim in the conclusion is not supported by the algorithm description.
minor comments (4)
  1. [General] All tables report point estimates without error bars or confidence intervals. Given the claim of 'no zero-shot accuracy loss,' at least a few seeds or calibration subsets are needed to assess whether differences of 0.1-2 points are meaningful.
  2. [Title page, §1] There are several typos and formatting errors: 'we proposedSparseSSM' should be 'we propose SparseSSM'; 'primarily' is misspelled as 'primally' in §2; Table 1 contains '22.3552.33' where a space between numbers is missing.
  3. [Fig. 3] The caption says the figure plots both zero-shot accuracy and WikiText perplexity, but the axis label only shows 'Accuracy'. If perplexity is intended, the figure should include a second axis or separate panel.
  4. [References] Some references are incomplete or malformed (e.g., author lists truncated in [3], [17], [18], and [48]). The bibliography needs to be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation is self-contained; Theorem 1's approximations are a correctness risk, not a circular reduction.

full rationale

SparseSSM's central derivation, Theorem 1 / Eq. (6), starts from the external OBS saliency definition and computes per-parameter importance by differentiating the SSM recurrence. The steps flagged by the skeptic—dropping the derivative of the backpropagated gradient with respect to A_log as a third-order effect, interpreting Eq. (17), and absorbing A^2 e^{2 delta A} into a global constant—are approximations that may be mathematically unsupported, and the final score A_log^2 times sum h^2 indeed coincides with a Wanda-style magnitude-times-activation heuristic. However, none of these steps defines the target result in terms of itself, fits a parameter to the reported perplexities or accuracies, or imports a conclusion through self-citation. The paper contains no load-bearing self-citation: the OBS and SparseGPT foundations are independent prior work, and the Mamba references are the original architecture papers rather than the authors' own uniqueness claims. Hyperparameters such as alpha and Nsample are selected experimentally, yet the comparison tables report external benchmark numbers rather than statements forced by the score's definition. The paper's claim of pruning 'without calibration' conflicts with its stated WikiText-2 calibration protocol, and the Theorem 1 proof may be incorrect or incomplete, but those are correctness and consistency concerns, not circularity. Accordingly, no circular step is identified.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central method depends on two hand-chosen hyperparameters (alpha, Nsample) and on three unproved mathematical assumptions: local quadraticity, diagonal Hessian, and neglect of gradient and third-order terms. No new entities are postulated. The heaviest burden is the diagonal-Hessian and constant-kappa approximation in Theorem 1, which is what converts OBS into the simple A^2 times sum h^2 score; if these fail, the theory is decorative even if the heuristic works.

free parameters (2)
  • FFN sparsity deviation interval alpha = 0.04
    Chosen by hand to balance sparsity across in_proj and out_proj modules; Eq. (7) uses it to define the allowable deviation interval.
  • Calibration sample count Nsample = 64
    Selected as the best trade-off between pruning quality and compute (Appendix B.1); affects all reported numbers.
assumptions (4)
  • standard math OBS loss is locally quadratic around the trained weights
    Invoked in Lemma 1 to justify the second-order Taylor expansion; standard in the pruning literature.
  • ad hoc to paper Hessian of the SSM loss is diagonal for Alog parameters
    Appendix A.1 Eq. (24) claims cross-derivatives vanish because A is diagonal, but outputs couple all parameters through the recurrent path, so off-diagonal Hessian terms are neglected without proof.
  • ad hoc to paper Third-order derivative terms and the gradient are approximated as a global constant kappa
    In Appendix A, Eqs. (12)-(18), the backpropagated gradient is treated as constant when differentiating again, and the parameter-dependent factor delta^2 e^{2 delta A} is absorbed into kappa with no quantitative justification.
  • domain assumption The 64 calibration segments represent the data distribution
    Calibration uses 64 segments of 2048 tokens from WikiText-2; the method assumes these states generalize to PTB, C4, and zero-shot tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot." pith.science (2026). https://pith.science/paper/AM6DNBUL

@misc{pith2026250609613,
  author       = {Pith},
  title        = {Pith review of: SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AM6DNBUL}},
  note         = {Machine review of arXiv:2506.09613}
}
read the original abstract

State-space language models such as Mamba match Transformer quality while permitting linear complexity inference, yet still comprise billions of parameters that hinder deployment. Existing one-shot pruning methods are tailored to attention blocks and fail to account for the time-shared and discretized state-transition matrix at the heart of the selective state-space module (SSM). In this paper, we introduce SparseSSM, the first training-free pruning framework that extends the classic optimal brain surgeon (OBS) framework to state space architectures. Our layer-wise algorithm (i) derives an approximate second-order saliency score that aggregates Hessian-trace information across time steps, (ii) incorporates a component sensitivity analysis to guide feed-forward network (FFN) pruning, which also sheds light on where redundancy resides in mamba architecture, (iii) can be easily extended to semi-structured and structured sparsity. Empirically, we prune 50% of SSM weights without fine-tuning and observe no zero-shot accuracy loss, achieving the current state-of-the-art pruning algorithm for Mamba-based LLMs.

Figures

Figures reproduced from arXiv: 2506.09613 by the authors.

Figure 1
Figure 1. Illustration of SparseSSM. The first row depicts the evolution of the diagonal parameter matrix Alog within the SSM module in Mamba, together with a schematic of the forward-propagation process. In the second row, the left panel shows the procedure for obtaining a mask from the Hessian estimate at a single time step (see Section 3.2), while the right panel presents our strategy for merging the masks across all time … view at source ↗
Figure 2
Figure 2. The Hessian matrices and correspond￾ing reconstruction errors for each module of the Mamba-370M FFN at a sparsity level of 50%. Dif￾ferent modules are represented by different colors. Motivated by these findings, we adopt the sensitivity-aware pruning framework, treating the in_proj and out_proj modules indepen￾dently. We use the trace of the Hessian matrix of the weights as the sensitivity score, and define the spa… view at source ↗
Figure 3
Figure 3. Performance of the full Mamba archi￾tecture at multiple sparsity levels by measuring zero-shot task accuracy and Wikitext perplexity We then apply one-shot unstructured pruning across all trainable weights except the input em￾bedding and output head. In this setting, each model typically incorporates an nn.Conv1d layer for feature preprocessing, in_proj and out_proj linear layers for dimensionality trans￾formation, … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Effects of calibration sample size and sparsity interval. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 34 canonical work pages

  1. [1]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, MarieAnne Lachaux, Timothe Lacroix, Baptiste Rozire, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  2. [2]

    Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, and Xi Victoria Lin et al. Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022

    arXiv 2022

  3. [3]

    Bloom: A 176b-parameter open-access multilingual language model, 2023

    BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´c, and Daniel Hesslow et al. Bloom: A 176b-parameter open-access multilingual language model, 2023

    work page 2023

  4. [4]

    LeCun, J

    Y . LeCun, J. S. Denker, and S. A. Solla. Optimal brain damage. InNeurIPS, 1990

    work page 1990

  5. [5]

    Hassibi and D

    B. Hassibi and D. G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. InNeurIPS, 1993

    1993

  6. [6]

    Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding

    Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. InICLR, 2016

    2016

  7. [7]

    Llm-pruner: On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. InNeurIPS, 2023

    work page 2023

  8. [8]

    SparseGPT: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. InICML, 2023

    work page 2023

Show all 52 references
  1. [9]

    Learning both weights and connections for efficient neural network

    Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural network. InNeurIPS, 2015

    2015

  2. [10]

    Pruning filters for efficient convnets

    Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. InICLR, 2017

    2017

  3. [11]

    Channel pruning for accelerating very deep neural networks

    Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. InICCV, 2017

    2017

  4. [12]

    Snip: Single-shot network pruning based on connection sensitivity

    Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. InICLR, 2019

    2019

  5. [13]

    Pruning neural networks without any data by iteratively conserving synaptic flow

    Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. InNeurIPS, 2020

    2020

  6. [14]

    Optimal brain compression: A framework for accurate post- training quantization and pruning

    Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post- training quantization and pruning. InICML, 2022

    2022

  7. [15]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  8. [16]

    Alps: Improved optimiza- tion for highly sparse one-shot pruning for large language models

    Xiang Meng, Kayhan Behdin, Haoyue Wang, and Rahul Mazumder. Alps: Improved optimiza- tion for highly sparse one-shot pruning for large language models. InNeurIPS, 2024

    2024

  9. [17]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  10. [18]

    Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. InICML, 2024

    2024

  11. [19]

    J. Zuo, M. Velikanov, D. E. Rhaiem, et al. Falcon mamba: The first competitive attention-free 7b language model.arXiv preprint arXiv:2410.05355, 2024

    2024 arXiv

  12. [20]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeurIPS, 2017. 10

    2017

  13. [21]

    Hippo: Recurrent memory with optimal polynomial projections

    Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Re. Hippo: Recurrent memory with optimal polynomial projections. InNeurIPS, 2020

    2020

  14. [22]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. InICLR, 2022

    2022

  15. [23]

    Smith, Andrew Warrington, and Scott Linderman

    Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. InICLR, 2023

    2023

  16. [24]

    Jamba: A hybrid transformer-mamba language model

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Ere...

    2024

  17. [25]

    Glorioso, Q

    P. Glorioso, Q. Anthony, Y . Tokpanov, et al. The zamba2 suite: Technical report.arXiv preprint arXiv:2411.15242, 2024

    2024 arXiv

  18. [26]

    Simba: Simplified mamba-based architecture for vision and multivariate time series.arXiv preprint arXiv:2403.15360, 2024

    Badri N Patro and Vijay S Agneeswaran. Simba: Simplified mamba-based architecture for vision and multivariate time series.arXiv preprint arXiv:2403.15360, 2024

    2024 arXiv

  19. [27]

    Hymba: A hybrid-head architecture for small language models.arXiv preprint arXiv:2411.13676, 2024

    Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabalesh- warkar, Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Lin, Jan Kautz, and Pavlo Molchanov. Hymba: A hybrid-head architecture for small language models.arXiv prep...

    2024 arXiv

  20. [28]

    Woodfisher: Efficient second-order approximations for model compression

    Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximations for model compression. InNeurIPS, 2020

    2020

  21. [29]

    Fast as chita: Neural network pruning with combinatorial optimization

    Riade Benbaki, Wenyu Chen, Xiang Meng, Hussein Hazimeh, Natalia Ponomareva, Zhe Zhao, and Rahul Mazumder. Fast as chita: Neural network pruning with combinatorial optimization. InICML, 2023

    2023

  22. [30]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InNAACL, 2019

    2019

  23. [31]

    Tycho F. A. van der Ouderaa, Markus Nagel, Mart van Baalen, Yuki M. Asano, and Tijmen Blankevoort. The llm surgeon. InICLR, 2024

    2024

  24. [32]

    Darwinlm: Evolutionary structured pruning of large language models.arXiv preprint arXiv:2502.07780, 2025

    Shengkun Tang, Oliver Sieberling, Eldar Kurtic, Zhiqiang Shen, and Dan Alistarh. Darwinlm: Evolutionary structured pruning of large language models.arXiv preprint arXiv:2502.07780, 2025

    2025 arXiv

  25. [33]

    Slimgpt: Layer-wise structured pruning for large language models

    Gui Ling, Ziyang Wang, Yuliang Yan, and Qingwen Liu. Slimgpt: Layer-wise structured pruning for large language models. InNeurIPS, 2024

    2024

  26. [34]

    Structured optimal brain pruning for large language models

    Jiateng Wei, Quan Lu, Ning Jiang, Siqi Li, Jingyang Xiang, Jun Chen, and Yong Liu. Structured optimal brain pruning for large language models. InNeurIPS, 2024

    2024

  27. [35]

    The iterative optimal brain surgeon: Faster sparse recovery by leveraging second-order information

    Diyuan Wu, Ionut-Vlad Modoranu, Mher Safaryan, Denis Kuznedelev, and Dan Alistarh. The iterative optimal brain surgeon: Faster sparse recovery by leveraging second-order information. InNeurIPS, 2024

    2024

  28. [36]

    The combinatorial brain surgeon: Pruning weights that cancel one another in neural networks

    Xin Yu, Thiago Serra, Srikumar Ramalingam, and Shandian Zhe. The combinatorial brain surgeon: Pruning weights that cancel one another in neural networks. InICML, 2022

    2022

  29. [37]

    Layer-adaptive state pruning for deep state space models

    Minsunu Kwak, Seungrok Moon, Joohwan Ko, and POOGYEON PARK. Layer-adaptive state pruning for deep state space models. InNeurIPS, 2024

    2024

  30. [38]

    Pablo Muñoz, Jinjie Yuan, and Nilesh Jain

    J. Pablo Muñoz, Jinjie Yuan, and Nilesh Jain. Mamba-shedder: Post-transformer compression for efficient selective structured state space models. InNAACL, 2025

    2025

  31. [39]

    Ghattas, M

    T. Ghattas, M. Hassid, and R. Schwartz. On pruning state-space llms.arXiv preprint arXiv:2502.18886, 2025. 11

    2025

  32. [40]

    Taghibakhshi, S

    A. Taghibakhshi, S. T. Sreenivas, S. Muralidharan, et al. Efficient hybrid language model compression through group-aware ssm pruning.arXiv preprint arXiv:2504.11409, 2025

    2025

  33. [41]

    P.J. Werbos. Backpropagation through time: what it does and how to do it.Proceedings of the IEEE, 1990

    1990

  34. [42]

    One-shot sensitivity-aware mixed sparsity pruning for large language models

    Hang Shao, Bei Liu, and Yanmin Qian. One-shot sensitivity-aware mixed sparsity pruning for large language models. InICASSP, 2024

    2024

  35. [43]

    Perplexity of fixed-length models

    Hugging Face. Perplexity of fixed-length models. 2022

    2022

  36. [44]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. InICLR, 2017

    2017

  37. [45]

    The penn treebank: Annotating predicate argument structure

    Mitch Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating predicate argument structure. InHuman Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, Ma...

    1994

  38. [46]

    Exploring the limits of transfer learning with a unified text-to-text transformer.JMLR, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.JMLR, 21(140):1–67, 2020

    2020

  39. [47]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. InAAAI, 2020

    2020

  40. [48]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InEMNLP, 2018

    2018

  41. [49]

    Winogrande: An adversarial winograd schema challenge at scale.arXiv preprint arXiv:1907.10641, 2019

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.arXiv preprint arXiv:1907.10641, 2019

    1907 arXiv

  42. [50]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  43. [51]

    mamba-minimal: A minimal pytorch implementation of mamba

    John Ma. mamba-minimal: A minimal pytorch implementation of mamba. https://github. com/johnma2006/mamba-minimal

  44. [52]

    Long short-term memory.Neural Computation, 9(8):1735–1780, 1997

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory.Neural Computation, 9(8):1735–1780, 1997. 12 A Proofs of Theorem 1 Proof.We begin at a trained network’s parameters, where Alog is near a local minimum of the loss L. In this setting, small perturbations of the par...

    1997

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.