Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Learning Fine-grained Parameter Sharing via Sparse Tensor Decomposition

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that fine-grained parameter sharing, implemented as a shared dense basis with sparse per-layer projections learned by block-wise activation reconstruction, compresses transformer MLPs by 50–75 percent with negligible…

desk verdict FiPS delivers strong ViT compression with a clean shared-basis/sparse-projection method, but the abstract's 'negligible' LLM perplexity claim is contradicted by the paper's own Table 3. read the letter →

arxiv 2411.09816 v6 pith:U2DWKK5B submitted 2024-11-14 cs.LG

Cem Üyük , Mike Lasby , Mohamed Yassin , Utku Evci , Yani Ioannou This is my paper · ORCID
classification cs.LG
keywords modelcompressionparametersharingtensordecompositionsparsityvisiontransformerslargelanguagemodelsSVDinitialization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that fine-grained parameter sharing is a practical way to compress transformers: instead of reusing whole blocks or pruning individual weights, it factorizes a group of MLP weight matrices into one shared dense basis and sparse per-layer projections. The claim is that this structure reduces the MLP parameter budget to 25–40 percent of the original for Vision Transformers while keeping top-1 accuracy within about a point, and to 50–60 percent for large language models with only small perplexity loss. If true, compression no longer has to be a per-layer decision: entire groups of layers can lean on a common low-dimensional substrate, and the same pipeline can be stacked with quantization. The paper also reports that this block-wise optimization is cheap enough to run on one GPU in under an hour for ViTs and a few hours for smaller LLMs.

What carries the argument

The central object is the factorization $W_i = U V_i$ over a group of MLP layers: $U$ is a dense shared basis in $\mathbb{R}^{d \times r}$ and $V_i$ are sparse layer-specific projections in $\mathbb{R}^{r \times p}$, produced by concatenating the $W_i$ along the long axis, applying truncated SVD, and gradually pruning the $V$ slices to about 75 percent sparsity. The basis $U$ stays dense and shared; the $V_i$ carry each layer's identity as a sparse combination of basis neurons. The work it does is to turn parameter sharing into a tensor-decomposition problem: sharing happens in the left factor, individual expression in the right factor, and block-wise L2 activation reconstruction supplies the training signal that makes the factorization track the original network's function rather than just its weights.

What would settle it

Run the DeiT-B 25%-budget FiPS recipe with the calibration set replaced by 2,560 images drawn from a single ImageNet superclass; if the resulting top-1 accuracy drops by more than about one point below the reported 80.64, the local reconstruction proxy is not tracking global accuracy.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that sparsity in the projection factor is what makes cross-layer neuron sharing work. Given weights $W_i$, FiPS concatenates them along the long axis into $W_s = [W_1; \dots; W_N]$, performs a truncated SVD $W_s = U\Sigma \hat{V}$ to get a shared basis $U$, and writes each layer as $W_i = U V_i$, where $V_i = \Sigma \hat{V}_i$ is sliced and then sparsified to about 75 percent. The $V$ factors are optimized by minimizing $\sum_i \lVert W_i X_i - U V_i X_i \rVert_2^2$ on calibration activations, with $U$ updated from the aggregated per-layer gradients. The paper's evidence is that this beats SVD-only and feature-mimicking baselines at matched budgets: at a 40 percent budget DeiT-B keeps 81.69 top-1 accuracy versus 81.85 for the original, and Gemma-2-9B at a 50 percent budget reaches 6.08 WikiText-2 perplexity versus 5.59 for the original. The central insight is that allowing each layer a sparse combination of shared neurons gives far more representational capacity per parameter than one-hot sharing, so sparsity is not a side constraint but the enabling mechanism.

Load-bearing premise

The load-bearing premise is that minimizing block-wise L2 activation error on a small calibration set (2,560 ImageNet images or 163,840 SlimPajama tokens) preserves full-model accuracy and perplexity for every tested model and budget.

Editorial extensions

If this is right

  • At 40–50 percent MLP budgets, DeiT-B and Swin-L remain within roughly 0.2 points of original ImageNet top-1 accuracy, so the shared basis is not merely a low-rank approximation but a functionally faithful replacement.
  • FiPS matches or exceeds the more expensive global feature-mimicking baseline (GFM) while doing only block-wise optimization, suggesting the local objective is sufficient for these budgets.
  • The same recipe transfers to LLMs: Gemma-2 and Llama-3 MLPs keep WikiText-2 perplexity within roughly 1 point at a 50 percent budget, and a compressed 9B/8B model can match the perplexity of an uncompressed smaller model.
  • FiPS composes with quantization: 3-bit FiPS on Gemma-2-2B at 8x compression achieves lower perplexity than 2-bit quantization-aware training alone, so sparsity-induced sharing and low-bit quantization capture different redundancies.
  • Transfer learning on CIFAR-100, Flowers102, Pets, and iNaturalist shows that compressed bases adapt to new tasks nearly as well as the original network, sometimes exceeding it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper only compresses MLP blocks; a natural extension is to apply the same shared-basis/sparse-projection factorization to attention projection matrices, where the paper's own conclusion says similar gains are likely but offers no measurements.
  • The use of a small calibration set means FiPS's reported numbers are conditional on that proxy; a stress test on out-of-distribution calibration data would clarify how robust the method is to domain shift, which the paper does not run.
  • Because $U$ is dense and shared while the $V_i$ are sparse, a deployment could keep the basis resident in fast memory and stream the per-layer projections; the paper's 2:4-sparsity latency benchmarks hint at this benefit but do not isolate cache reuse of the shared basis.
  • If the block-wise L2 objective is the right proxy, the method should extend to other layer types and architectures with alternating projections and sparsity schedules; the paper only tests two ViT families and four LLM families.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes FiPS, a post-training compression method for transformer MLPs that combines cross-layer parameter sharing, low-rank tensor decomposition, and sparsity. A group of MLP weight matrices is concatenated and decomposed as W_i = U V_i, where U is a shared dense basis and V_i are sparse layer-specific projections; the factors are initialized by truncated SVD and then optimized by block-wise L2 activation reconstruction (Eq. 1), with optional end-to-end fine-tuning. Experiments on DeiT-B and Swin-L report ImageNet-1k top-1 accuracy within 1 point of the original at 25-50% parameter budgets, and experiments on Gemma-2 and Llama-3 models report WikiText-2 perplexity at 50-60% MLP parameter budgets, with large gains over plain SVD baselines where those are reported.

Significance. If the central claims hold, fine-grained cross-layer parameter sharing is a practical and previously underexplored compression axis that can be stacked with pruning and quantization. The paper has several concrete strengths: it ships code, gives detailed hyperparameters and block-group configurations, includes latency and memory profiling for structured sparsity, and ablates initialization, sparsity method, calibration size, and compression groups. The ViT results, in particular, are internally coherent and show a consistent advantage over AAFM/GFM baselines. However, the LLM-perplexity claim of 'negligible degradation' is not supported by the reported numbers for most models, and the abstract promises QAT results that do not appear in the full text.

major comments (3)
  1. [§4.2, Table 3; §1; §7] The claim that FiPS compresses LLM MLPs by 40-50% 'while maintaining perplexity with negligible degradation' is contradicted by the paper's own Table 3. At a 50% parameter budget, Llama-3.2-3B rises from 6.19 to 9.70 WikiText-2 perplexity (+57% relative) and Llama-3.1-8B rises from 4.75 to 7.11 (+50% relative); only the two Gemma-2 models are within roughly 10%. The central framing in the introduction and conclusion must be revised to reflect that Llama compression at the advertised budget incurs substantial perplexity loss, or additional evidence (e.g., larger calibration sets, multiple seeds, or evaluation on additional held-out corpora) must be provided to show that these are conservative artifacts rather than genuine limitations.
  2. [Abstract; §1; §4.2] The abstract claims that 'Combined with Quantization-Aware Training (QAT), 3-bit FiPS on Gemma-2-2B achieves lower perplexity than 2-bit QAT alone while matching the same 8x compression' and also claims downstream-benchmark improvements, but no QAT experiments, 3-bit results, or downstream-benchmark tables appear anywhere in the full text. This is a missing-results problem, not a wording issue: either the experiments must be added or the abstract must be reduced to what the paper actually demonstrates.
  3. [§4.1, §4.2, §5] All headline results are reported as single runs without error bars or seed variance, while the method's hyperparameters (learning rates, calibration length, sparsity schedule, and block-group size) are selected using the same validation split or test set used for the reported numbers. Given the sensitivity to calibration size and training length shown in Figure 4c, and the large Llama perplexity gaps in Table 3, the paper should either report variance across multiple runs or explicitly state that all numbers are point estimates from a single hyperparameter selection; as written, the reader cannot distinguish systematic degradation from hyperparameter or calibration sensitivity.
minor comments (4)
  1. [§5, Table 4] The text states that 'RigL consistently outperforms both Dense and Static Sparsity across parameter budgets ranging from 10% to 50%,' but Table 4 shows Static Sparsity slightly outperforming RigL at the 40% and 50% budgets (81.48 vs. 81.50 and 81.70 vs. 81.65). The sentence should be corrected to reflect the actual comparisons.
  2. [§5] The paragraph beginning 'Sparsity Distribution and MSE-loss' is duplicated verbatim; one copy should be removed.
  3. [Table 3] The SVD baseline is reported only for the two smaller models (Gemma-2-2B and Llama-3.2-3B), so the claimed advantage of FiPS over SVD is not directly demonstrated for Gemma-2-9B and Llama-3.1-8B; adding those columns would strengthen the comparison.
  4. [§2, parameter-budget definition] The definition of parameter budget in §2 is phrased awkwardly ('the fraction of retained parameters after truncated SVD minus the nonzero parameters preserved in the decomposed factors after sparsification'); it should be reworded to state clearly that the budget is the ratio of total nonzero parameters in the decomposed representation to the original module's parameter count.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FiPS is an empirical compression method whose headline results are held-out benchmark measurements, not fitted inputs or self-citation chains.

full rationale

The derivation chain runs from the low-rank form W_i = U V_i, initialized by truncated SVD of concatenated MLP weights, to block-wise activation reconstruction, Eq. (1). The reported accuracy and perplexity numbers are external benchmark measurements on ImageNet-1k and held-out WikiText-2, with calibration sets (2,560 ImageNet images; 163,840 SlimPajama tokens) separate from the evaluation sets. No fitted quantity is renamed as a prediction: the parameter budget is defined as the count of retained nonzero parameters after decomposition and sparsification, and task performance is measured on unseen data. Self-citations (RigL, GradMax, and the Lasby et al. structured-sparsity baseline) appear as optimization tools or comparison baselines in ablations and do not generate the headline numbers; none is invoked as a uniqueness theorem or as the justification for the central compression claim. The grouping of adjacent blocks is motivated empirically by measured MSE and post-compression accuracy, not by definition. The skeptical note's concerns, such as Table 3 showing larger perplexity degradation for Llama models than the abstract's 'negligible degradation' wording suggests, and the abstract mentioning 3-bit QAT and downstream-benchmark results not present in the body, are correctness and support issues rather than circularity. Because the core claims are benchmarked against held-out data and existing baselines, the paper is self-contained with respect to circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The empirical claims rest mainly on a handful of tuned hyperparameters (sparsity level, group size, calibration length, learning rates, growth scaling) rather than on a fitted physical constant. The central assumptions are local-reconstruction transfer, long-axis concatenation, adjacent-block grouping, and representational sufficiency of sparse shared bases. The paper introduces no new physical or categorical entities: U and V are standard linear-algebra factors. The main missing safeguards are held-out model selection, error bars, and an ablation isolating cross-block sharing from per-layer sparse low-rank compression.

free parameters (6)
  • Target sparsity level of V factors = 75%
    Set by sensitivity analysis in Fig. 4b; 75% is best for DeiT-B, lower sparsity reduces rank, and higher sparsity hurts reconstruction.
  • Parameter-sharing group size = 4 blocks (DeiT-B), 2/6 groups (Swin-L), 2/6 (Gemma-2), 7 (Llama-3.2-3B), 8 (Llama-3.1-8B)
    Chosen from reconstruction and accuracy experiments in Fig. 3; no automatic or principled selection rule is given.
  • Calibration length = 20 epochs over 20 batches for ViTs; 40 epochs over 163,840 tokens for LLMs
    Selected via Fig. 4c as the most efficient setting within 0.25% of the best; this reduces compute but may under-tune the factors.
  • Neuron-growth scaling tau = not reported numerically, tuned by sweep
    Appendix A.1.2 says tau is treated as a hyperparameter and chosen by a sweep for hybrid initialization when rank exceeds d.
  • Learning rates = GMP/RigL 1e-3, Dense 1.25e-4, static 2.5e-4, transfer 7.5e-6 to 1e-4, LLM 4e-6 to 3.125e-5
    Selected from 8-12 point logarithmic sweeps; single values are reported per model with no error bars.
  • Sparsity schedule and growth ratios = GMP: 25% initial to 75% final; RigL: growth 0.1/0.05, DeltaT=50
    Defaults from cited prior work plus hand tuning; no ablation varying the schedule is reported.
assumptions (5)
  • domain assumption Block-wise L2 activation matching on a small calibration set preserves downstream task accuracy and perplexity.
    Eq. (1) is the entire second phase of FiPS; no proof or held-out analysis links local reconstruction error to global task performance.
  • domain assumption Concatenating FC weights along the long axis and sharing among adjacent blocks is near-optimal.
    Chosen from Fig. 2c and Fig. 3a on DeiT-B, then applied to all ViTs and LLMs without further validation.
  • domain assumption Representing neurons as sparse linear combinations of a shared basis retains task-relevant information at the tested budgets.
    Underlies W = UV; validated only empirically per model and budget.
  • standard math Truncated SVD provides a suitable initialization for U and V.
    Uses the Eckart-Young theorem for optimal rank-r reconstruction in Frobenius norm, but optimality in that norm does not imply optimal task accuracy.
  • domain assumption Calibration data (ImageNet subset, SlimPajama subset) is representative of the target evaluation distribution.
    Compression is tuned on these sets; evaluation uses ImageNet validation and WikiText-2, which come from different distributions than the calibration data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Fine-grained Parameter Sharing via Sparse Tensor Decomposition." pith.science (2026). https://pith.science/paper/U2DWKK5B

@misc{pith2026241109816,
  author       = {Pith},
  title        = {Pith review of: Learning Fine-grained Parameter Sharing via Sparse Tensor Decomposition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U2DWKK5B}},
  note         = {Machine review of arXiv:2411.09816}
}
read the original abstract

Large neural networks achieve state-of-the-art performance on many tasks, yet their sheer size hinders deployment on resource-constrained devices. Among existing compression approaches, cross-layer parameter sharing remains relatively unexplored for transformer models. In this paper, we introduce Fine-grained Parameter Sharing (FiPS), a unified framework for compressing transformer Multi-Layer Perceptrons (MLPs) that combines cross-block parameter sharing, low-rank factorization, and sparsity in a single optimization. FiPS concatenates MLP weight matrices across a group of transformer blocks and factorizes them into a shared basis and sparse, layer-specific projection matrices. Both factors are initialized via singular value decomposition (SVD) and jointly optimized by block-wise reconstruction error minimization. FiPS compresses Vision Transformers (ViTs) by up to 33% with less than 1% top-1 accuracy loss on ImageNet-1k, and by up to 57% when combined with fine-tuning. It also compresses Large Language Models (LLMs) by up to 20% while outperforming existing SVD-based methods in perplexity and downstream benchmarks at matched compression. Combined with Quantization-Aware Training (QAT), 3-bit FiPS on Gemma-2-2B achieves lower perplexity than 2-bit QAT alone while matching the same 8x compression. These results establish fine-grained parameter sharing as a practical and effective approach for transformer MLP compression.

Figures

Figures reproduced from arXiv: 2411.09816 by the authors.

Figure 1
Figure 1. Fine-grained Parameter Sharing (FiPS) employs a shared basis for the sparse weight matrices in the Fully Connected (FC) layers of Multi-Layer Perceptron (MLP) modules across transformer blocks. This approach is detailed in § 2 and § 3. Sharing parameters across multiple neural network layers can theoretically reduce memory usage and improve cache ef￾ficiency, potentially accelerating execution. Building on this idea… view at source ↗
Figure 2
Figure 2. Initial Experiments. Reconstruction error when inducing sparsity on different factors of the low-rank decomposition for (a) FC-1 and (b) FC-2 layers under a 25% parameter budget. Higher sparsity in larger factors (V ) enables a higher rank and lower reconstruction error. (c) Mean reconstruction error across four FC layers under various parameter sharing schemes. See § 2.2 for a description of each strategy. concaten… view at source ↗
Figure 3
Figure 3. Parameter Sharing Groups. (a top) Mean squared error (MSE) increases when sharing U across different MLP modules, with red squares indicating that sharing adjacent modules enhances reconstruction. (a bottom) MSE for compressing individual MLP modules, demonstrating that sharing U among consecutive layers typically results in the lowest error. (b) For a fixed parameter budget, the rank of the shared basis U stabilize… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: DeiT-B Ablations and Sensitivity Analysis.(a) Analysis of key components in the FiPS algorithm, including Random Initialization (RI), Local Pruning (LP), Global Prun￾ing (GP), and Scaling Vectors (SV); (b) Impact of varying sparsity levels on performance; (c) Influence…
Figure 5
Figure 5. Figure 5: DeiT-B Global Sparsity Analysis. (a) Average sparsity at the end of the training shows that more parameters are allocated to later modules. (b) A strong correlation is observed between the MSE reported in Figure 3a and the parameter distribution identified by FiPS. bet…
Figure 6
Figure 6. Figure 6: Parameter Sharing Through Sparse Tensor Decomposition. A group of FC layers are concatenated along the larger dimension, p, and decomposed into two matrices: a shared basis, U, and a sparse projection matrix, which is then sliced up respectively for each layer [PITH_F…
Figure 7
Figure 7. Figure 7: DeiT-B Inference Latency and Memory Benchmarks. (a) End-to-end latency of 2:4 sparse FiPS on an NVIDIA A4000 for batch sizes ranging from 1 to 64. FiPS with a 22% parameter budget exhibits a 25% latency improvement over the original network above batch sizes of 8. (b) …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. On Information Geometry and Iterative Optimization in Model Compression: Operator Factorization

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Compression methods can be seen as information projections, and iterative soft rank reduction, not the projection choice, dominates performance after fine-tuning, with a convergence proof for iterative singular value ...

Reference graph

Works this paper leans on

35 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Intrinsic dimensionality explains the effectiveness of language model fine-tuning, 2020

    Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, 2020. URL https://arxiv.org/abs/2012.13255

    arXiv 2020

  2. [2]

    Net2net: Accelerating learning via knowledge transfer, 2016

    Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer, 2016. URL https://arxiv.org/abs/1511.05641

    arXiv 2016

  3. [3]

    A survey of model compression and acceleration for deep neural networks, 2020

    Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks, 2020. URL https://arxiv.org/abs/1710.09282

    arXiv 2020

  4. [4]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 248--255. IEEE, 2009

    work page 2009

  5. [5]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/abs/2010.11929

    arXiv 2021

  6. [6]

    Structured Multi-Hashing for Model Compression

    Elad Eban, Yair Movshovitz-Attias, Hao Wu, Mark Sandler, Andrew Poon, Yerlan Idelbayev, and Miguel A. Carreira-Perpinan. Structured multi-hashing for model compression, 2019. URL https://arxiv.org/abs/1911.11177

    work page Pith review arXiv 2019

  7. [7]

    Rigging the lottery: Making all tickets winners, 2021

    Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners, 2021. URL https://arxiv.org/abs/1911.11134

    arXiv 2021

  8. [8]

    Gradmax: Growing neural networks using gradient information, 2022

    Utku Evci, Bart van Merriënboer, Thomas Unterthiner, Max Vladymyrov, and Fabian Pedregosa. Gradmax: Growing neural networks using gradient information, 2022. URL https://arxiv.org/abs/2201.05125

    arXiv 2022

Show all 35 references
  1. [9]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015. URL https://arxiv.org/abs/1502.01852

    2015 arXiv

  2. [10]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In Advances in Neural Information Processing Systems (NIPS) Deep Learning and Representation Learning Workshop, 2015

    2015

  3. [11]

    Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks, 2021

    Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks, 2021. URL https://arxiv.org/abs/2102.00554

    2021 arXiv

  4. [12]

    Kolda and Brett W

    Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. SIAM Rev., 51: 0 455--500, 2009. URL https://api.semanticscholar.org/CorpusID:16074195

    2009

  5. [13]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, 2009

    2009

  6. [14]

    How to prune your language model: Recovering accuracy on the "sparsity may cry'' benchmark, 2023

    Eldar Kurtic, Torsten Hoefler, and Dan Alistarh. How to prune your language model: Recovering accuracy on the "sparsity may cry'' benchmark, 2023. URL https://arxiv.org/abs/2312.13547

    2023 arXiv

  7. [15]

    Albert: A lite bert for self-supervised learning of language representations, 2020

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations, 2020. URL https://arxiv.org/abs/1909.11942

    2020 arXiv

  8. [16]

    Sparsimony - Dynamic sparse training and pruning algorithms for PyTorch , September 2024

    Mike Lasby. Sparsimony - Dynamic sparse training and pruning algorithms for PyTorch , September 2024. URL https://github.com/mklasby/sparsimony

    2024

  9. [17]

    Understanding parameter sharing in transformers, 2023

    Ye Lin, Mingxuan Wang, Zhexi Zhang, Xiaohui Wang, Tong Xiao, and Jingbo Zhu. Understanding parameter sharing in transformers, 2023. URL https://arxiv.org/abs/2306.09380

    2023 arXiv

  10. [18]

    Dora: Weight-decomposed low-rank adaptation, 2024

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation, 2024. URL https://arxiv.org/abs/2402.09353

    2024 arXiv

  11. [19]

    Swin transformer: Hierarchical vision transformer using shifted windows, 2021

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021. URL https://arxiv.org/abs/2103.14030

    2021 arXiv

  12. [20]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

    2019 arXiv

  13. [21]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, pp.\ 722--729. IEEE, 2008

    2008

  14. [22]

    T-basis: a compact representation for neural networks, 2021

    Anton Obukhov, Maxim Rakhuba, Stamatios Georgoulis, Menelaos Kanakis, Dengxin Dai, and Luc Van Gool. T-basis: a compact representation for neural networks, 2021. URL https://arxiv.org/abs/2007.06631

    2021 arXiv

  15. [23]

    Parkhi, Andrea Vedaldi, Andrew Zisserman, and C

    Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. IEEE Conference on Computer Vision and Pattern Recognition, 2012. The Oxford-IIIT Pet Dataset

    2012

  16. [24]

    Sparse connection and pruning in large dynamic artificial neural networks

    Nikko Ström. Sparse connection and pruning in large dynamic artificial neural networks. In 5th European Conference on Speech Communication and Technology (Eurospeech 1997), pp.\ 2807--2810, 1997. doi:10.21437/Eurospeech.1997-708

    1997 doi

  17. [25]

    Lessons on parameter sharing across layers in transformers, 2023

    Sho Takase and Shun Kiyono. Lessons on parameter sharing across layers in transformers, 2023. URL https://arxiv.org/abs/2104.06022

    2023 arXiv

  18. [26]

    Evaluating pruning methods

    Georg Thimm and Emile Fiesler. Evaluating pruning methods. In International Symosium on Artifical Neural Networks, 1995. URL https://api.semanticscholar.org/CorpusID:11075297

    1995

  19. [27]

    Training data-efficient image transformers & distillation through attention, 2021

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention, 2021. URL https://arxiv.org/abs/2012.12877

    2021 arXiv

  20. [28]

    The inaturalist species classification and detection dataset

    Grant Van Horn, Oisin Mac Aodha, Trevor Marquis, Steve Su, Mona Haghighi, Jason Baldridge, Subhransu Maji, and Pietro Perona. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), ...

    2018

  21. [29]

    Compressingtransformers

    Hao Yu and Jianxin Wu. Compressingtransformers. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Inte...

    2023 doi

  22. [30]

    Minivit: Compressing vision transformers with weight multiplexing, 2022

    Jinnian Zhang, Houwen Peng, Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Minivit: Compressing vision transformers with weight multiplexing, 2022. URL https://arxiv.org/abs/2204.07154

    2022 arXiv

  23. [31]

    To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017

    Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017. URL https://arxiv.org/abs/1710.01878

    2017 arXiv

  24. [32]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  25. [33]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  26. [34]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  27. [35]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.