Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A whitened low-rank projection provably attains the best possible per-layer reconstruction error at any rank, and a closed-form distortion formula turns budgeted, fine-tuning-free compression into a single fast pass, across linear and…

desk verdict Solid per-layer theory, practical pipeline, but rank allocator rests on an energy proxy whose only formal end-to-end bound is admitted loose; worth serious review. read the letter →

arxiv 2509.25136 v3 pith:LMLCYQOS submitted 2025-09-29 cs.LG

David González-Martínez This is my paper · ORCID
classification cs.LG MSC 68T0715A18
keywords low-rankfactorizationfine-tuning-freecompressionactivation-awareuncenteredwhiteningbudgetedrankallocationgroupedconvolutionsvisiontransformerssingularvaluedecomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that neural networks can be compressed without any fine-tuning while still retaining near-optimal accuracy, by factorizing each layer around the data that actually flows through it rather than around the raw weights. Its central claim is a provable optimality result: truncating a whitened version of the layer's weight matrix reproduces, exactly, the lowest possible per-layer output reconstruction error at any rank, for fully connected, convolutional, and grouped-convolutional layers alike. A closed-form formula for that error turns rank selection into a cheap optimization, letting the user specify a FLOP or parameter-count budget and have per-layer ranks assigned in a single fast pass. If right, the method removes both fine-tuning and expensive search from the compression pipeline, making structured compression practical on commodity hardware.

What carries the argument

The load-bearing identity is the whitening relation: for a layer input $X$ with uncentered whitening matrix $M$ (defined by $M^T X^T X M = N I_R$, where $R$ is the rank of $X$), the squared singular values of $XW$ equal $N$ times the squared singular values of $M^+ W$. This makes truncating $M^+ W$ by SVD and then un-whitening equivalent to directly truncating the layer's outputs, the Eckart–Young–Mirsky optimal approximation. The same relation yields the closed-form distortion formula of Theorem 2. Around this identity the paper builds the $(\mathcal{O},\mathcal{I},\mathcal{P})$-expressible layer representation (a layer written as $\mathcal{O}(\mathcal{I}(X)\mathcal{P}(W))$, a batch of $G$ matrix multiplications between reshaped activations and reshaped weights), which is what extends the method to grouped convolutions, and a Lagrangian-relaxation rank allocator that solves a multiple-choice knapsack for per-layer ranks under a cost budget.

What would settle it

Compress a fixed network at a fixed budget (for instance ResNet-50 at 50% of parameters) and compare BALF's energy-maximal rank allocation with an exhaustive search over rank allocations at the identical cost; if any allocation with lower total retained energy yields materially higher top-1 accuracy, the energy proxy fails. A cheaper version: locate one layer whose high-energy truncation barely hurts accuracy while a low-energy truncation elsewhere collapses it, which would directly contradict the allocator's objective.

Watch

Extended reading notes

Core claim

BALF's central claim is that activation-aware low-rank factorization can be made both general and provably optimal. For any layer that can be written as a (possibly grouped) matrix multiplication sandwiched between reshape and permute operations—fully connected layers, standard convolutions, grouped convolutions, and transformer linear layers—the paper defines the projection $T^{\mathrm{AA}}_P(W) = \mathcal{P}^{-1}(M\, T^{\mathrm{SVD}}_P(M^+ \mathcal{P}(W)))$, where $M$ is an uncentered whitening matrix for the layer input's second-moment matrix. Theorem 1 states that this projection is optimal in the sense of Definition 2: its output distortion equals that of the best possible rank-$P$ approximation to the layer's outputs, so no rank-$P$ factorization can do better in per-layer output error. Theorem 2 gives the closed-form distortion $\frac{N}{B}\sum_{g=1}^{G}\sum_{i=P+1}^{U}\sigma_{g,i}^2$, the tail squared singular values of $M^+\mathcal{P}(W)$, which turns rank selection into a quantity measured for free during compression. On top of this, the paper proposes a Lagrangian relaxation of a multiple-choice knapsack problem that allocates per-layer ranks to maximize total retained activation energy under an explicit FLOP or parameter budget, and shows experimentally across CNNs and vision transformers that the full pipeline beats SVD-based baselines at matched complexity.

Load-bearing premise

The budget allocator relies on the assumption that preserving each layer's retained output energy is a trustworthy stand-in for preserving the model's accuracy, even though the paper's formal bridge between per-layer and whole-model error is admitted to be typically loose in deep networks.

Editorial extensions

If this is right

  • Any $(\mathcal{O},\mathcal{I},\mathcal{P})$-expressible layer—fully connected, standard and grouped convolutional, and transformer linear layers—can be compressed with provably minimal per-layer output distortion at a chosen rank, so the guarantee is not limited to linear layers.
  • Because the per-layer distortion is available in closed form from the SVD already computed during compression, rank selection costs under 0.2 seconds and never queries the model, so a user-specified FLOP or parameter budget is met in a single run.
  • At matched parameter or FLOP budgets, the activation-aware projection plus budgeted allocator yields higher accuracy than SVD-with-energy truncation in the fine-tuning-free regime, with a 45% FLOP reduction on ResNeXt-101 costing about 1 percentage point of top-1 accuracy.
  • Whiteners estimated on clean training data carry over to corrupted inputs: the additional accuracy drop under CIFAR-10-C distribution shift is typically below 2.5 points, so the compressed model does not disproportionately lose robustness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the closed-form distortion is exact per layer and cheap, BALF could serve as the distortion oracle inside expensive search-based compressors, turning hours of neural-architecture-search over ranks into a single post-hoc re-ranking step; the author hints at such a combination with SVD-NAS as future work.
  • The energy-maximizing allocator is only one objective on top of the exact per-layer oracle: a principled alternative, untested in the paper, would weight each layer's distortion by the Lipschitz-chain factors of Theorem 3, which the author noted was not improved by ad hoc variance or entropy weightings.
  • The framework's only input statistics are second moments of layer activations, so nothing in the method is vision-specific; the same whitening machinery should apply to transformer or convolutional encoders in other modalities, an extension the paper does not run.
  • The shared per-group rank constraint for grouped convolutions is an implementation choice, not a mathematical necessity; relaxing it at the cost of bucketed kernels could extract additional compression from groups with unbalanced spectra.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The manuscript introduces BALF, a fine-tuning-free model-compression pipeline that frames linear, convolutional, and grouped-convolutional layers as (O,I,P)-expressible maps and factorizes them with activation-aware whitening. The main theoretical results are Theorem 1, claiming optimality of the proposed projection T^AA_P in the sense of per-layer output distortion, and Theorem 2, giving a closed-form expression (N/B) * sum_g sum_{i>P} sigma_{g,i}^2 for that distortion. The paper then formulates rank allocation under FLOP or parameter budgets as a multiple-choice knapsack problem, solves it with a Lagrangian relaxation, and evaluates the resulting pipeline on CIFAR-10 and ImageNet across CNNs and vision transformers. The reported experiments show BALF generally outperforming SVD-based baselines and being competitive with other fine-tuning-free pruning and factorization methods.

Significance. If the experimental results hold, BALF is a practically useful contribution to the fine-tuning-free compression literature. The framework unifies SVD-LLM-style activation-aware factorization and extends it to grouped convolutions; Theorems 1 and 2 are proved in Appendix I with elementary linear algebra and appear correct. The closed-form distortion formula provides an interpretable and essentially free per-layer distortion measure, and the Lagrangian allocator is cheap, taking less than 0.2 seconds in the reported runs. The code release and reproducibility statement are notable strengths. The main caveat is that the connection from per-layer distortion or retained energy to end-to-end accuracy is empirical; the theory alone does not imply that the rank-allocation objective maximizes final accuracy.

major comments (2)
  1. [Section 4.3, equation after 'Energy-based selection'] The allocator maximizes sum_l E_l(P_l), where E_l(P_l) is the normalized retained energy, i.e., the fraction of the layer's own singular-value energy that is kept. Theorem 2, however, bounds the unnormalized per-layer activation distortion (N_l/B) * sum_g sum_{i>P_l} sigma_{g,i}^2, and layers with larger output scale N_l * ||f_l(X_{l-1}; W_l)||^2_F contribute more to total output distortion. Equal weighting of normalized E_l therefore has no theoretical justification as a surrogate for minimizing total distortion, and the statement in Appendix C that unnormalized or weighted alternatives 'did not improve' is given without any quantitative comparison. This is a load-bearing gap because budgeted rank allocation is one of the two claimed contributions. A quantitative ablation reporting total distortion and accuracy for unnormalized distortion, output-scale-weighted energy, and the chosen normalized energy should be added.
  2. [Section 4.4 and Algorithm 1] Theorems 1 and 2 are derived under the assumption that layer l receives the original calibration activations X_{l-1}. In the deployed compressed network, upstream layers are also replaced, so layer l receives \tilde{X}_{l-1}, which is not equal to X_{l-1}; the whitening matrices and singular values computed on X_{l-1} therefore no longer certify the distortion of the actual network. The only formal end-to-end bound, Theorem 3 (formalized as Theorem 4 in Appendix D), is explicitly acknowledged to be 'typically loose in practical deep networks'. Consequently, the theory does not guarantee that a rank allocation maximizing normalized energy computed on original activations preserves accuracy. The paper should measure the actual per-layer distortion on the sequentially compressed network, or otherwise quantify the distribution-shift gap, in order to connect the theoretical distortion guarantees to the reported end-to-end accuracy.
minor comments (4)
  1. [Section 5, Figure 2 and Table 1] No error bars or multiple-seed variance are reported for the experimental results, including the CIFAR-10 models that are trained by the authors; since several reported differences between methods are only a few accuracy points, standard deviations over at least three runs for a subset of configurations would strengthen the comparison.
  2. [Appendix C.1, Algorithm 2] The text claims that the optimal Lagrange multiplier is guaranteed to reside in the interval and that the interval shrinks exponentially, but the algorithm only bisects I times and returns the feasible selection at lambda_max; the approximation gap to the multiple-choice knapsack optimum is not characterized. A brief empirical or theoretical characterization of this gap would be helpful.
  3. [Table 1 and Appendix E.3] Several baseline numbers are taken from prior papers, and some are extracted from figures with WebPlotDigitizer; a brief note on the exact evaluation protocols and preprocessing used by those baselines would help the reader judge the fairness of the comparison.
  4. [Section 4.3 and Appendix C] The symbol U_l is used in the main text for the total number of singular values per group and in Appendix C for an upper bound on the rank; please unify the notation to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorems 1 and 2 are self-contained derivations from the whitening definition and the Eckart–Young–Mirsky theorem; the rank allocator's energy proxy is a stated heuristic limitation, not a fitted prediction.

full rationale

The paper's central formal claims, Theorems 1 and 2, are derived in Appendix I from first principles. Theorem 2 is an algebraic identity: using the defining property M^T X^T X M = N I_R of the uncentered whitening matrix, the per-layer output distortion of T^AA_P reduces exactly to (N/B) times the tail sum of squared singular values of M^+P(W). This is not a fitted constant or a restatement of the desired conclusion; it follows directly from the stated definitions and the SVD truncation. Theorem 1 then proves optimality by showing that the same expression equals the Eckart–Young–Mirsky error for the layer outputs, again through the whitening property rather than by construction. The rank allocator in Section 4.3 maximizes normalized retained energy as a proxy objective for choosing ranks; the paper does not claim this proxy provably maximizes end-to-end accuracy. Indeed, the only formal cumulative link, Theorem 3/4, is explicitly described as 'typically loose in practical deep networks', and Appendix C reports that alternative objectives did not improve results. These are acknowledged limitations and empirical heuristics, which are correctness/robustness concerns rather than circular reasoning. The paper builds on SVD-LLM (Wang et al., 2025c), but the optimality argument for BALF is self-contained and does not reduce to that citation; there is no load-bearing self-citation, and the information-geometric discussion in Appendix H is explicitly post-hoc and interpretive. Overall, no derivation step is equivalent to its inputs by definition, and no fitted parameter is renamed as a prediction; the honest finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The central mathematical claim (Theorems 1 and 2) rests only on standard linear algebra and the definition of the uncentered whitening matrix. The experimental effectiveness relies on the domain assumptions listed above, mostly the calibration-transfer assumption and the energy proxy. No new physical or model entities are introduced.

free parameters (1)
  • solver iterations I = 300
    Hand-set hyperparameter for the Lagrangian relaxation (Algorithm 2). Performance stabilizes quickly (Figure 4), but it is not fit to accuracy and slightly affects solution quality.
assumptions (6)
  • domain assumption Calibration data is i.i.d. and representative of the test distribution.
    Section 4.1 and Appendix F.2 assume the calibration set (subset of training data) is representative of the test distribution so that second-moment statistics transfer to unseen data.
  • domain assumption The target layers are (O,I,P)-expressible.
    Definition 1 restricts the framework to layers expressible as reshape/permute around a linear map and batched matmul. The paper assumes the relevant layers (fully connected, ungrouped and grouped convolutions) satisfy this.
  • standard math Activation functions have finite Lipschitz constants and linear maps have finite operator norms.
    Theorem 3/4 assumes element-wise activations have Lipschitz constants A_l and linear maps I_l have operator norms B_l to bound cumulative output distortion.
  • ad hoc to paper Maximizing normalized retained activation energy is a valid proxy for preserving task accuracy.
    Section 4.3 defines the allocator objective as maximizing per-layer retained energy; the paper provides no theoretical guarantee linking this to final accuracy and notes Theorem 3 is loose. This is an empirical assumption.
  • ad hoc to paper The Lagrangian relaxation approximates the knapsack optimum well enough.
    The rank allocation problem is NP-hard; the paper solves it with a bisection-based Lagrangian heuristic (Algorithm 2) and argues it works well in practice, but it is not guaranteed to find the optimal allocation.
  • domain assumption Shared per-group rank for grouped convolutions is a practical necessity.
    Appendix A.3.1 justifies forcing the same rank across groups for implementation efficiency; this may be suboptimal compared with per-group ranks, but is presented as a practical necessity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression." pith.science (2026). https://pith.science/paper/LMLCYQOS

@misc{pith2026250925136,
  author       = {Pith},
  title        = {Pith review of: BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LMLCYQOS}},
  note         = {Machine review of arXiv:2509.25136}
}
read the original abstract

Activation-aware low-rank factorization techniques yield strong compression results but are generally confined to linear layers, while existing whitening-based theory typically makes an implicit full-rank assumption on activations. We introduce a layer representation framework that extends activation-aware factorization beyond linear layers, including standard and grouped convolutions. Within this framework, our whitening-based formulation is more general than prior ones, naturally covering rank-deficient activations, and yields an optimal low-rank projection that attains the reconstruction error of the best low-rank approximation to layer activations. The resulting singular spectrum provides a closed-form per-layer distortion proxy, which we use to allocate per-layer ranks under explicit FLOP or parameter-count budgets via a Lagrangian relaxation with negligible overhead. Together, these components form BALF, an end-to-end pipeline for efficient vision model compression. Across CNNs and vision transformers on CIFAR-10 and ImageNet-1K, BALF generally achieves higher accuracy than SVD-based factorization baselines at matched FLOP or parameter count targets and remains competitive with other fine-tuning-free compression techniques.

Figures

Figures reproduced from arXiv: 2509.25136 by the authors.

Figure 1
Figure 1. Normalized output distortion (in terms of squared Frobenius norm) with T AA P [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Sweep over different compression ratios with different decomposition methods on differ [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Calibration dataset size vs. accuracy over different compression ra￾tios (in terms of parameter counts) on ResNet-50. Other results. On CIFAR-10-C (Hendrycks & Diet￾terich, 2019), BALF remains robust to distribution shift: when whitening matrices were estimated on the standard CIFAR-10 training set and evaluation was performed on CIFAR-10-C, the additional accuracy drop was typically less than 2.5 pp. Moreover, we f… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The effect of the number of iterations on CIFAR-10 models. We fix a compression target [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: Calibration size vs. performance across models. Hyperparameters are fixed to those of [PITH_FULL_IMAGE:figures/full_fig_p023_5.png]
Figure 6
Figure 6. Figure 6: Compression-accuracy curves on ResNet-20 over different compression ratios on CIFAR [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]
Figure 7
Figure 7. Figure 7: Throughput vs. FLOPs retained on ViT-B/16 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 FLOPs ratio retained 500 600 700 800 Throughput (items/sec) ResNet-50 Batch size batch=8 batch=16 batch=32 [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]
Figure 9
Figure 9. Figure 9: Additional figures with experimental results that did not appear in the main text. [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SLORR: Simple and Efficient In-Training Low-Rank Regularization

    cs.LG 2026-07 accept novelty 6.0 of 10

    A stateless, SVD-free regularizer approximates polar factors to induce low-rank weight structure during training, enabling better post-training compression of vision models and LLMs at under 8% overhead.

Reference graph

Works this paper leans on

61 extracted references · 32 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Compressing pre-trained language models by matrix decomposition

    Matan Ben Noach and Yoav Goldberg. Compressing pre-trained language models by matrix decomposition. In Kam-Fai Wong, Kevin Knight, and Hua Wu (eds.), Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp.\ 884--889, Suzho...

    work page doi:10.18653/v1/2020.aacl-main.88 2020

  3. [3]

    High Performance Convolutional Neural Networks for Document Processing

    Kumar Chellapilla, Sidd Puri, and Patrice Simard. High Performance Convolutional Neural Networks for Document Processing . In Guy Lorette (ed.), Tenth International Workshop on Frontiers in Handwriting Recognition , La Baule (France), October 2006. Universit \'e de Rennes 1 , Suvisoft . URL https://inria.hal.science/inria-00112631. http://www.suvisoft.com

    work page 2006

  4. [4]

    Drone: Data-aware low-rank compression for large nlp models

    Patrick Chen, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. Drone: Data-aware low-rank compression for large nlp models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 29321--29334. Curran Associates, Inc., 2021. URL https://proceedings.neurips.c...

    work page 2021

  5. [5]

    Going beyond neural network feature similarity: The network feature complexity and its interpretation using category theory

    Yiting Chen, Zhanpeng Zhou, and Junchi Yan. Going beyond neural network feature similarity: The network feature complexity and its interpretation using category theory. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=4bSQ3lsfEV

    work page 2024

  6. [6]

    Parseval networks: improving robustness to adversarial examples

    Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: improving robustness to adversarial examples. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, pp.\ 854–863. JMLR.org, 2017

    work page 2017

  7. [7]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 248--255, 2009. doi:10.1109/CVPR.2009.5206848

    arXiv 2009

  8. [8]

    Exploiting linear structure within convolutional networks for efficient evaluation

    Emily Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS'14, pp.\ 1269–1277, Cambridge, MA, USA, 2014. MIT Press

    work page 2014

Show all 61 references
  1. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  2. [10]

    Pulp: A linear programming toolkit for python

    Iain Dunning, Stuart Mitchell, and Michael O'Sullivan. Pulp: A linear programming toolkit for python. Technical report, Department of Engineering Science, The University of Auckland, September 2011. URL https://optimization-online.org/?p=11731

    2011

  3. [11]

    Optimal brain compression: A framework for accurate post-training quantization and pruning

    Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openrevi...

    2022

  4. [12]

    OPTQ : Accurate quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ : Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=tcbBPnfwxS

    2023

  5. [13]

    Mahoney, and Kurt Keutzer

    Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. CoRR, abs/2103.13630, 2021. URL https://arxiv.org/abs/2103.13630

    2021 arXiv

  6. [14]

    Golub and C.F

    G.H. Golub and C.F. Van Loan. Matrix Computations. Johns Hopkins Studies in the Mathematical Sciences. Johns Hopkins University Press, 2013. ISBN 9781421407944. URL https://books.google.es/books?id=X5YfsuCWpxMC

    2013

  7. [15]

    Feature variance ratio-guided channel pruning for deep convolutional network acceleration

    Junjie He, Bohua Chen, Yinzhang Ding, and Dongxiao Li. Feature variance ratio-guided channel pruning for deep convolutional network acceleration. In Hiroshi Ishikawa, Cheng-Lin Liu, Tomas Pajdla, and Jianbo Shi (eds.), Computer Vision -- ACCV 2020, pp.\ 170--186, Cham, 2021. S...

    2020

  8. [16]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. URL http://arxiv.org/abs/1512.03385

    2015 arXiv

  9. [17]

    Benchmarking neural network robustness to common corruptions and perturbations

    Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019

    2019

  10. [18]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  11. [19]

    Magma: Enabling exascale performance with accelerated blas and lapack for diverse gpu architectures

    Michael Heroux, Ahmad Abdelfattah, Natalie Beams, Robert Carson, Pieter Ghysels, Tzanio Kolev, Thomas Stitt, Arturo Vargas, Stanimire Tomov, and Jack Dongarra. Magma: Enabling exascale performance with accelerated blas and lapack for diverse gpu architectures. Int. J. High Per...

    2024 doi

  12. [20]

    Sparsity in deep learning: pruning and growth for efficient inference and training in neural networks

    Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res., 22 0 (1), January 2021. ISSN 1532-4435

    2021

  13. [21]

    Dynamic low-rank estimation for transformer-based language models

    Ting Hua, Xiao Li, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. Dynamic low-rank estimation for transformer-based language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 927...

    2023 doi

  14. [22]

    Carreira-Perpiñán

    Yerlan Idelbayev and Miguel Á. Carreira-Perpiñán. Low-rank compression of neural nets: Learning the rank of each layer. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 8046--8056, 2020. doi:10.1109/CVPR42600.2020.00807

    2020

  15. [23]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Le...

    2015

  16. [24]

    Quantization and training of neural networks for efficient integer-arithmetic-only inference

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision an...

    2018

  17. [25]

    Speeding up convolutional neural networks with low rank expansions

    Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. ArXiv, abs/1405.3866, 2014. URL https://api.semanticscholar.org/CorpusID:17864746

    2014 arXiv

  18. [26]

    Optimal whitening and decorrelation

    Agnan Kessy, Alex Lewin, and Korbinian Strimmer. Optimal whitening and decorrelation. The American Statistician, 72 0 (4): 0 309--314, 2018. doi:10.1080/00031305.2016.1277159. URL https://doi.org/10.1080/00031305.2016.1277159

    2018 arXiv

  19. [27]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL https://api.semanticscholar.org/CorpusID:18268744

    2009

  20. [28]

    Imagenet classification with deep convolutional neural networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 20...

    2012

  21. [29]

    Oseledets, and Victor S

    Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, I. Oseledets, and Victor S. Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. CoRR, abs/1412.6553, 2014. URL https://api.semanticscholar.org/CorpusID:15002492

    2014 arXiv

  22. [30]

    Deep learning

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 0 (7553): 0 436--444, 2015. doi:10.1038/nature14539. URL https://doi.org/10.1038/nature14539

    2015 doi

  23. [31]

    Training-free restoration of pruned neural networks, 2025

    Keonho Lee, Minsoo Kim, and Dong-Wan Choi. Training-free restoration of pruned neural networks, 2025. URL https://arxiv.org/abs/2502.08474

    2025 arXiv

  24. [32]

    Compressing neural networks: Towards determining the optimal layer-wise decomposition

    Lucas Liebenwein, Alaa Maalouf, Dan Feldman, and Daniela Rus. Compressing neural networks: Towards determining the optimal layer-wise decomposition. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021....

    2021

  25. [33]

    TorchVision: PyTorch's Computer Vision library , November 2016

    TorchVision maintainers and contributors. TorchVision: PyTorch's Computer Vision library , November 2016. URL https://github.com/pytorch/vision

    2016

  26. [34]

    TVSP rune - pruning non-discriminative filters via total variation separability of intermediate representations without fine tuning

    Chaitanya Murti, Tanay Narshana, and Chiranjib Bhattacharyya. TVSP rune - pruning non-discriminative filters via total variation separability of intermediate representations without fine tuning. In The Eleventh International Conference on Learning Representations, 2023. URL ht...

    2023

  27. [35]

    Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning, ICML'10, pp.\ 807–814, Madison, WI, USA, 2010. Omnipress. ISBN 9781605589077

    2010

  28. [36]

    DFPC : Data flow driven pruning of coupled channels without data

    Tanay Narshana, Chaitanya Murti, and Chiranjib Bhattacharyya. DFPC : Data flow driven pruning of coupled channels without data. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=mhnHqRqcjYU

    2023

  29. [37]

    cuSOLVER Library

    NVIDIA . cuSOLVER Library . NVIDIA, 2025. URL https://docs.nvidia.com/cuda/cusolver/. Release 13.0

    2025

  30. [38]

    PyTorch: an imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K\" o pf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu...

    2019

  31. [39]

    Stable low-rank tensor decomposition for compression of convolutional neural network

    Anh-Huy Phan, Konstantin Sobolev, Konstantin Sozykin, Dmitry Ermilov, Julia Gusak, Petr Tichavsk \'y , Valeriy Glukhov, Ivan Oseledets, and Andrzej Cichocki. Stable low-rank tensor decomposition for compression of convolutional neural network. In Andrea Vedaldi, Horst Bischof,...

    2020

  32. [40]

    Dobi- SVD : Differentiable SVD for LLM compression and some new perspectives

    Wang Qinsi, Jinghan Ke, Masayoshi Tomizuka, Kurt Keutzer, and Chenfeng Xu. Dobi- SVD : Differentiable SVD for LLM compression and some new perspectives. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=kws76i5XB8

    2025

  33. [41]

    Mobilenetv2: Inverted residuals and linear bottlenecks

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018

    2018

  34. [42]

    On information geometry and iterative optimization in model compression: Operator factorization, 2025

    Zakhar Shumaylov, Vasileios Tsiaras, and Yannis Stylianou. On information geometry and iterative optimization in model compression: Operator factorization, 2025. URL https://arxiv.org/abs/2507.09428

    2025 arXiv

  35. [43]

    Elrt: Efficient low-rank training for compact convolutional neural networks, 2024

    Yang Sui, Miao Yin, Yu Gong, Jinqi Xiao, Huy Phan, and Bo Yuan. Elrt: Efficient low-rank training for compact convolutional neural networks, 2024. URL https://arxiv.org/abs/2401.10341

    2024 arXiv

  36. [44]

    Solution methods for the multiple-choice knapsack problem and their applications

    Tibor Szkaliczki. Solution methods for the multiple-choice knapsack problem and their applications. Mathematics, 13 0 (7), 2025. ISSN 2227-7390. doi:10.3390/math13071097. URL https://www.mdpi.com/2227-7390/13/7/1097

    2025 doi

  37. [45]

    Training data-efficient image transformers &; distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers &; distillation through attention. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machi...

    2021

  38. [46]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Info...

    2017

  39. [47]

    Forget the data and fine-tuning! just fold the network to compress

    Dong Wang, Haris S iki \'c , Lothar Thiele, and Olga Saukh. Forget the data and fine-tuning! just fold the network to compress. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=W2Wkp9MQsF

    2025

  40. [48]

    Practical network acceleration with tiny sets, 2023

    Guo-Hua Wang and Jianxin Wu. Practical network acceleration with tiny sets, 2023. URL https://arxiv.org/abs/2202.07861

    2023 arXiv

  41. [49]

    SVD-LLM V2 : Optimizing singular value truncation for large language model compression, 2025 b

    Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. SVD-LLM V2 : Optimizing singular value truncation for large language model compression, 2025 b . URL https://arxiv.org/abs/2503.12340

    2025 arXiv

  42. [50]

    SVD - LLM : Truncation-aware singular value decomposition for large language model compression

    Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD - LLM : Truncation-aware singular value decomposition for large language model compression. In The Thirteenth International Conference on Learning Representations, 2025 c . URL https://openreview.net/forum?id=LNYIUouhdt

    2025

  43. [51]

    PyTorch Image Models

    Ross Wightman. PyTorch Image Models . URL https://github.com/huggingface/pytorch-image-models

  44. [52]

    Aggregated residual transformations for deep neural networks

    Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. arXiv preprint arXiv:1611.05431, 2016

    2016 arXiv

  45. [53]

    Gtp-vit: Efficient vision transformers via graph-based token propagation, 2024

    Xuwei Xu, Sen Wang, Yudong Chen, Yanping Zheng, Zhewei Wei, and Jiajun Liu. Gtp-vit: Efficient vision transformers via graph-based token propagation, 2024. URL https://arxiv.org/abs/2311.03035

    2024 arXiv

  46. [54]

    Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification

    Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification. CoRR, abs/2004.09031, 2020. URL https://arxiv.org/abs/2004.09031

    2004 arXiv

  47. [55]

    Towards efficient tensor decomposition-based dnn model compression with optimization framework

    Miao Yin, Yang Sui, Siyu Liao, and Bo Yuan. Towards efficient tensor decomposition-based dnn model compression with optimization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 10674--10683, June 2021

    2021

  48. [56]

    SVD-NAS : Coupling low-rank approximation and neural architecture search, 2022

    Zhewen Yu and Christos-Savvas Bouganis. SVD-NAS : Coupling low-rank approximation and neural architecture search, 2022. URL https://arxiv.org/abs/2208.10404

    2022 arXiv

  49. [57]

    Asvd: Activation-aware singular value decomposition for compressing large language models, 2025

    Zhihang Yuan, Yuzhang Shang, Yue Song, Dawei Yang, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models, 2025. URL https://arxiv.org/abs/2312.05821

    2025 arXiv

  50. [58]

    Dense vision transformer compression with few samples

    Hanxiao Zhang, Yifan Zhou, and Guo-Hua Wang. Dense vision transformer compression with few samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 15825--15834, 2024

    2024

  51. [59]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  52. [60]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  53. [61]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.