Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models

Aaron Thomas; Cheng Zhang; Jianyi Cheng; Xinxin Liu; Xitong Gao; Yiren Zhao

REVIEW 3 major objections 5 minor 46 references

Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A static mask chosen once by gradient magnitude is the best way to pick sparse fine-tuning weights, beating LoRA and PiSSA across NLP benchmarks.

desk verdict Useful broad benchmark of salience metrics for sparse PEFT, but the headline claim of consistent superiority over LoRA is weakened by a learning-rate mismatch in the MMLU experiments. read the letter →

arxiv 2412.13488 v2 pith:BNZU6LLB submitted 2024-12-18 cs.CL cs.AI

Xinxin Liu , Aaron Thomas , Cheng Zhang , Jianyi Cheng , Yiren Zhao , Xitong Gao This is my paper · ORCID

classification cs.CLcs.AI

keywords parameter-efficientfine-tuningsparsesaliencemetricsstaticmasksdynamicLoRAlargelanguagemodelsgradient-basedselection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks which salience metric should choose the few trainable weights in sparsity-based parameter-efficient fine-tuning (SPEFT), and whether the chosen mask should stay fixed or be refreshed during training. It evaluates eight first- and second-order salience metrics across GLUE, MMLU, GSM8K, and code benchmarks, and finds that plain first-order gradient information is the most reliable selector. It also finds that a static mask computed once before training performs as well as dynamic re-masking, at a fraction of the compute. The paper claims that a simple gradient-based, static SPEFT variant consistently outperforms LoRA and PiSSA when all methods use the same number of trainable parameters, and proposes it as a strong baseline for PEFT.

What carries the argument

The central object is the sparse adapter $\theta_{\mathrm{sp}}$ with binary mask $\tau \in \{0,1\}^{d_1 \times d_2}$, reparameterizing each layer as $\theta = \theta_0 + \theta_{\mathrm{sp}}$ with $1[\theta_{\mathrm{sp}} \neq 0] = \tau$ and $|\tau|_0 \le \rho$. The mask is built by computing one of eight salience scores $S(\theta)$ and keeping the top $\rho$ fraction of values, either globally across all layers or locally within each layer. The mechanism that carries the argument is the gradient salience score $\partial \ell/\partial \theta$ combined with a static mask: it requires only a short pre-training gradient estimation (64 steps of batch size 16), and during fine-tuning only the masked nonzero entries of $\theta_{\mathrm{sp}}$ are updated. This zero-cost selection and the absence of mask updates are what make the method both simple and efficient.

What would settle it

Re-run the GLUE, MMLU, and GSM8K comparisons with the best learning rate for each method found by the same sweep protocol; if LoRA or PiSSA then matches or beats gradient-based static SPEFT on average, the paper's central claim would collapse.

Watch

Extended reading notes

Core claim

The paper's central discovery is that, in sparse PEFT, where a frozen pretrained weight matrix $\theta_0$ is augmented by a trainable sparse matrix $\theta_{\mathrm{sp}}$ whose nonzero entries are fixed by a mask $\tau$, the simple gradient $\partial \ell/\partial \theta$ is the most reliable salience score for constructing the mask. It outperforms magnitude, SNIP, FORCE, Taylor-FO, SynFlow, GRaSP, and Fisher information on the evaluated tasks. The paper further argues that static masking, which selects the top-$\rho$ gradient values once before training and never changes them, is sufficient: dynamic masking refreshed every 1000 steps offers no substantial benefit and adds computational overhead from re-ranking and optimizer reinitialization. On this basis, the paper claims that gradient-based static SPEFT consistently beats LoRA and PiSSA on GLUE, MMLU, GSM8K, HumanEval, and MBPP with matched trainable-parameter counts.

Load-bearing premise

The comparison depends on LoRA and PiSSA having received learning-rate tuning as careful as the sparse adapters, but Appendix A shows different learning rates for the two families on the same datasets, so a less-tuned baseline could explain part of the reported gap.

Editorial extensions

If this is right

A static gradient-based sparse adapter can serve as a simple default PEFT baseline, offering a cheaper alternative to dynamic masking and a competitive or better alternative to LoRA and PiSSA at equal parameter budgets.
Second-order salience metrics such as Fisher information and GRaSP do not pay for themselves in SPEFT, so first-order gradient information should be the starting point for mask construction.
Dynamic masking's extra computational cost, from periodic mask recomputation and optimizer reinitialization, is not justified by its empirical performance.
The advantage of gradient-based SPEFT appears to widen on harder generation and reasoning tasks, such as GSM8K math word problems and code generation, where it beats LoRA by large margins.
As hardware support for sparse matrix multiplication matures, the efficiency advantage of static SPEFT should grow, since the fixed mask is known in advance and can be compiled into sparse kernels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves open whether the same ordering of salience metrics holds when sparse adapters are combined with other efficiency techniques, such as quantization of the frozen base model; a natural extension would freeze the gradient-chosen mask and quantize the residual $\theta_{\mathrm{sp}}$.
If the result transfers beyond language models, it suggests that parameter location, not low-rank structure, is what makes PEFT work; this could be tested by comparing gradient-selected sparse adapters against full-rank adapters with the same number of trainable parameters.
The paper's finding that global and local sparsity give comparable results hints that per-layer budget allocation matters less than the choice of salience signal, which could be probed with adaptive per-layer budgets.
Since the static mask is chosen before training, it could be combined with mixtures of sparse experts or multi-task sparse mask transfer, as the paper notes, to reuse one mask across related tasks.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's real contribution is the systematic comparison: eight salience metrics, static vs dynamic masks, across six model families and several task types. That benchmark is worth having, and the main practical message—simple gradient-based static SPEFT is a strong, cheap baseline—is persuasive. The study is large and mostly careful: repeated runs for smaller models, ablations on masking strategy, overhead measurements, and released code.

The soft spots are real but not fatal. First, the abstract says gradient-based SPEFT "consistently outperforms" other methods, but the paper's own tables undercut that. Fisher-Info wins three of six GLUE tasks on BERT-base, Taylor-FO matches or beats gradient on code (Table 12), and GRaSP is close on GSM8K. "Consistently" is doing too much work; the data support "competitive and often best."

More serious is the hyperparameter mismatch the stress-test flags. In Appendix A Table 6, on the same MMLU datasets, SPEFT uses 1e-5 or 5e-6 while LoRA uses 5e-5—a tenfold difference—even though Section 4 claims "kept the remaining hyperparameters always the same." This affects exactly the rows where the reported margin is a fraction of a point (e.g., Gemma2-2b OASST2: 53.11 vs 52.59). That is a legitimate confound, and the authors do not acknowledge it or document a symmetric best-vs-best tuning procedure for the larger models. Notably, the GLUE, MetaMathQA, and CodeFeedback tables share learning rates, so the core comparison partially holds; the problem is specifically the MMLU evidence that carries the headline claim. Also, large-model results are single runs, so sub-point differences are within noise.

What the paper does well: it assembles a useful public benchmark, tests static vs dynamic masking and global vs local sparsity, and honestly notes in the limitations that SPEFT seemed less sensitive to hyperparameters than LoRA—an observation worth following up.

Who is this for? Anyone doing PEFT and wanting a baseline that is simpler than LoRA and often matches it. It deserves a serious referee, but I would send it back for major revision: retune the low-rank baselines with a symmetric sweep or report a range over learning rates, and soften the consistency claim to match the tables. The MMLU comparison needs to be re-run or explicitly caveated. If that is fixed, this becomes a solid reference point for the field.

Referee Report

3 major / 5 minor

Summary. The paper studies sparsity-based parameter-efficient fine-tuning (SPEFT), where a sparse additive weight matrix is fine-tuned while pretrained weights are frozen. It evaluates eight salience metrics (magnitude, gradient, SNIP, FORCE, Taylor-FO, SynFlow, GRaSP, Fisher information) for constructing the initial sparsity mask, compares static versus dynamically updated masks and global versus local sparsity, and benchmarks against LoRA and PiSSA on GLUE (OPT, BERT, RoBERTa), MMLU (Gemma2-2b, Qwen2-7b), GSM8K (Gemma2-2b on MetaMathQA), and HumanEval/MBPP (Llama3-8b on CodeFeedback). The main claim is that a simple gradient-based, static SPEFT variant consistently outperforms other PEFT methods and is a strong baseline. The paper also argues that dynamic masking provides no substantial benefit over static masking. It releases open-source code.

Significance. If the central claim held, the contribution would be practically valuable: a static, gradient-magnitude mask selected once at initialization gives a cheaper PEFT alternative that matches or exceeds low-rank adapters on several benchmarks. The study's systematic comparison of salience proxies, the inclusion of both static and dynamic masking, and the release of code are strengths, as are the repeated runs with standard deviations for the smaller models. However, the headline claim of consistent superiority is not fully supported by the paper's own tables, and the large-model comparison contains a learning-rate asymmetry that must be resolved. The paper's contribution is real but needs refinement.

major comments (3)

[§4, Appendix A Table 6] The claim that 'kept the remaining hyperparameters always the same' (Section 4) is contradicted by Table 6, where Gemma2-2b and Qwen2-7b use SPEFT learning rates of 1e-5 or 5e-6 while LoRA uses 5e-5 on the same Alpaca/OASST2 setups. These 10x differences occur on the exact MMLU rows in Table 2 that support the large-model advantage, with margins of only 0.04–1.19 points (e.g., Gemma2-2b OASST2: 53.11 vs 52.59; Qwen2-7b OASST2: 70.55 vs 70.42). Because no symmetric best-vs-best tuning procedure is documented for these larger models, I cannot tell whether the reported gain is a method effect or a learning-rate artifact. The concern is not that per-method learning rates differ; it is that the paper neither acknowledges the difference nor shows that the comparison is apples-to-apples. This must be resolved before the central 'consistently outperforms' claim is acceptable.
[Table 12] In the code-generation experiments, the data do not support the abstract's claim that gradient-based SPEFT 'consistently outperforms other fine-tuning methods.' Taylor-FO achieves a higher average (49.40 vs 49.39) and a higher HumanEval score (49.39 vs 48.78), while SNIP is close (48.98). Section 4.2's statement that 'the lead by sparse adapters widens' is therefore unsupported on HumanEval/MBPP. The claim should be restricted to the benchmarks where it holds, or qualified with statistical significance tests.
[Tables 1, 9, 10] The word 'consistently' is also too strong on GLUE. While gradient-based SPEFT has the best average on several models, Fisher-Info wins more tasks on BERT-base (Table 1: 3 vs 2) and PiSSA wins more on RoBERTa-base (Table 9: 3 vs 2). The sentence 'gradient-based SPEFT has the best average accuracy, higher than LoRA and PiSSA' is fine, but the introduction and abstract overstate per-task consistency. Please temper the language or add a paired-test analysis.

minor comments (5)

[Abstract] 'Results are on par with the best alternatives' is difficult to reconcile with the later sentence 'consistently outperforms other fine-tuning methods'; please choose one or state the qualification.
[Table 12] The FORCE row reports an average of 34.91, which is inconsistent with HumanEval 46.95 and MBPP 50.4 (the average should be about 48.68); this appears to be a typo and should be corrected.
[Table 2] Table 2 does not include the PiSSA baseline, although Section 4 states that PiSSA is included as a low-rank baseline across models; please clarify whether PiSSA was run on the large models or why it is omitted.
[Section 4.4] The statement that salience estimation is 'negligible' refers to a one-time cost for static masks; for dynamic masks the cost is incurred every I steps. Please make this distinction explicit in the efficiency discussion.
[Section 8] The admission that SPEFT appeared less sensitive to hyperparameters than LoRA was not explored is important; given the learning-rate differences in Table 6, this unexplored observation should at least be mentioned in the main text as a caveat.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the SPEFT comparison is an empirical benchmark study with no derivation that reduces to its inputs.

full rationale

The paper's claims are empirical rather than derivational. The sparse mask is constructed by Eq. (2) as the top-ρ entries of a salience metric, and Algorithm 1 then trains only those entries; the headline result that gradient-based static SPEFT outperforms LoRA and PiSSA is a measured benchmark outcome, not a quantity implied by the definition of the mask or by the optimization objective. The eight salience metrics in Section 3.2 are standard, externally defined metrics and are not defined in terms of the final accuracy, so there is no self-definitional loop. The static-versus-dynamic comparison is also an empirical observation: dynamic masks are updated by re-ranking the same salience metric, and the paper reports that this does not help, which is a contingent result rather than an identity. No load-bearing self-citations appear: references to DiffPruning, FishMASK, Fish-DIP, and LF-SFT are background attributions, and the paper invokes no uniqueness theorem or prior-work ansatz to force its mask choice. The Appendix A hyperparameter mismatch (e.g., Gemma2-2b sparse SPEFT at 1e-5/5e-6 versus LoRA at 5e-5 on the same MMLU datasets) is a potential fairness confound in the empirical comparison, but it is not circularity: even if the low-rank baselines were undertuned, the SPEFT numbers would still be measured rather than derived from the method's own inputs. That concern belongs under correctness or experimental-fairness risk, not under the circularity rubric. Accordingly, no circular step is identified and the score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its central comparison depends on a set of hyperparameters: sparsity density, learning rates, mask update interval, and salience estimation budget. The main domain assumption is that low-cost proxies computed at initialization select weights that remain useful after training.

free parameters (4)

sparsity density rho = 0.18% to 0.97% per model
The fraction of trainable weights is set per model to match LoRA's trainable-parameter count; all SPEFT comparisons depend on this match.
learning rate for SPEFT = 5e-6 to 4e-4 depending on dataset
Learning rates were tuned separately for sparse and low-rank methods in Tables 5-8, so performance differences may partly reflect learning-rate choices.
mask update interval I = 1000 steps
Dynamic masks are refreshed every 1000 training steps; the conclusion that dynamic masking adds no benefit is specific to this schedule.
salience estimation budget = 1024 examples (64 steps x batch 16)
All data-aware metrics are estimated on this fixed sample; the paper assumes this is sufficient for stable masks.

assumptions (3)

domain assumption Salience measured at initialization predicts the value of a weight for final fine-tuned performance.
The SPEFT method selects the top-rho weights by proxy and never re-evaluates in the static case; if proxies are uninformative, the method fails. Invoked in Section 3.2 and Algorithm 1.
domain assumption The Fisher information can be approximated by the squared gradient (dL/dtheta)^2, ignoring off-diagonal Hessian terms.
Used to define the Fisher-Info metric in Section 3.2; this is a standard but unproven approximation in the pruning literature.
domain assumption Gradient estimates computed on 1024 examples are stable enough to select masks.
Section 4.4 states 64 steps of batch size 16 are used; no stability analysis is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models." pith.science (2026). https://pith.science/paper/BNZU6LLB

@misc{pith2026241213488,
  author       = {Pith},
  title        = {Pith review of: Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BNZU6LLB}},
  note         = {Machine review of arXiv:2412.13488}
}

read the original abstract

Parameter-Efficient Fine-Tuning (PEFT) has gained prominence through low-rank adaptation methods like LoRA. In this paper, we focus on sparsity-based PEFT (SPEFT), which introduces trainable sparse adaptations to the weight matrices in the model, offering greater flexibility in selecting fine-tuned parameters compared to low-rank methods. We conduct the first systematic evaluation of salience metrics for SPEFT, inspired by zero-cost NAS proxies, and identify simple gradient-based metrics is reliable, and results are on par with the best alternatives, offering both computational efficiency and robust performance. Additionally, we compare static and dynamic masking strategies, finding that static masking, which predetermines non-zero entries before training, delivers efficiency without sacrificing performance, while dynamic masking offers no substantial benefits. Across NLP tasks, a simple gradient-based, static SPEFT consistently outperforms other fine-tuning methods for LLMs, providing a simple yet effective baseline for SPEFT. Our work challenges the notion that complexity is necessary for effective PEFT, while our open-source framework establishes a reproducible benchmark for future research, which is available at [https://github.com/0-ml/speft].

Figures

Figures reproduced from arXiv: 2412.13488 by the authors.

**Figure 1.** Comparison between LoRA (Hu et al., 2021) and SPEFT. LoRA freezes pretrained weights θ0 and updates the low-rank terms A and B, while SPEFT adopts zero-cost proxies to build a sparse adapter θsp, to update the weight elements that contribute most to the downstream task. of training sufficient, or is a dynamically updated pruning mask preferable? In this paper, we systematically re-examine the design principles for … view at source ↗

**Figure 2.** Varing the number of trainable parameters [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 14 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

https://www.amd.com/en/products/accelerators/instinct/mi300.html

AMD I nstinct MI 300 S eries A ccelerators. https://www.amd.com/en/products/accelerators/instinct/mi300.html. Accessed: 2024-03-03

work page 2024
[4]

Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vuli \'c . 2021. Composable sparse fine-tuning for cross-lingual transfer. arXiv preprint arXiv:2110.07560

arXiv 2021
[5]

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

arXiv 2021
[6]

Yoshua Bengio, Nicholas L \'e onard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432

arXiv 2013
[7]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

2020
[8]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

arXiv 2021

Show all 46 references

[9]

Jack Choquette. 2023. NVIDIA H opper H 100 GPU : S caling P erformance. IEEE Micro, (3):9--17

2023
[10]

Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. 2021. NVIDIA A 100 tensor core GPU : P erformance and innovation. IEEE Micro, (2):29--35

2021
[11]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . ...

2021 arXiv
[12]

Sarkar Snigdha Sarathi Das, Ranran Haoran Zhang, Peng Shi, Wenpeng Yin, and Rui Zhang. 2023. Unified low-resource sequence labeling by sample-aware dynamic sparse finetuning. arXiv preprint arXiv:2311.03748

2023 arXiv
[13]

Pau de Jorge, Amartya Sanyal, Harkirat Behl, Philip Torr, Gr \'e gory Rogez, and Puneet K. Dokania. 2021. https://openreview.net/forum?id=9GsFOUyUPi Progressive skeletonization: Trimming more fat from a network at initialization . In International Conference on Learning Repres...

2021
[14]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://arxiv.org/abs/1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding . Preprint, arXiv:1810.04805

2019 arXiv
[15]

Dolan and Chris Brockett

William B. Dolan and Chris Brockett. 2005. https://aclanthology.org/I05-5002/ Automatically constructing a corpus of sentential paraphrases . In Proceedings of the Third International Workshop on Paraphrasing ( IWP 2005)

2005
[16]

Jonathan Frankle and Michael Carbin. 2019. https://openreview.net/forum?id=rJl-b3RcF7 The lottery ticket hypothesis: Finding sparse, trainable neural networks

2019
[17]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

2024 arXiv
[18]

Demi Guo, Alexander M Rush, and Yoon Kim. 2020. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463

2020 arXiv
[19]

Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149

2015 arXiv
[20]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . Preprint, arXiv:2009.03300

2021 arXiv
[21]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

2021 arXiv
[22]

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Binarized neural networks. Advances in neural information processing systems, 29

2016
[23]

Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and A...

2023 arXiv
[24]

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942

2019 arXiv
[25]

Yann LeCun, John Denker, and Sara Solla. 1989. Optimal brain damage. Advances in neural information processing systems, 2

1989
[26]

Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. 2019 a . https://openreview.net/forum?id=B1VZqjAcYX SNIP : Single-shot network pruning based on connection sensitivity . In International Conference on Learning Representations

2019
[27]

Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. 2019 b . https://arxiv.org/abs/1810.02340 Snip: Single-shot network pruning based on connection sensitivity . Preprint, arXiv:1810.02340

2019 arXiv
[28]

Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. 2021. Group fisher pruning for practical network compression. In International Conference on Machine Learning, pages 7021--7032. PMLR

2021
[29]

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

2019 arXiv
[30]

Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. 2023. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of Machine Learning Research, 24(253):1--15

2023
[31]

Fanxu Meng, Zhaohui Wang, and Muhan Zhang. 2024. https://arxiv.org/abs/2404.02948 Pissa: Principal singular values and singular vectors adaptation of large language models . Preprint, arXiv:2404.02948

2024 arXiv
[32]

Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. 2019. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11264--11272

2019
[33]

Justin Selig. 2022. The cerebras software development kit: A technical overview. Technical Report, Cerebras

2022
[34]

Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. 2020. Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint arXiv:2008.09777, 4:14

2020 arXiv
[35]

Tianxiang Sun, Yunfan Shao, Xiaonan Li, Pengfei Liu, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020. Learning sparse sharing architectures for multiple tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8936--8943

2020
[36]

Yi-Lin Sung, Varun Nair, and Colin A Raffel. 2021. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193--24205

2021
[37]

Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. 2020. https://openreview.net/forum?id=HJgKShEtvS Pruning neural networks without any data by iteratively conserving synaptic flow . In International Conference on Learning Representations

2020
[38]

Hashimoto

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

2023
[39]

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

2024 arXiv
[40]

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. https://arxiv.org/abs/1804.07461 Glue: A multi-task benchmark and analysis platform for natural language understanding . Preprint, arXiv:1804.07461

2019 arXiv
[41]

Chaoqi Wang, Guodong Zhang, and Roger Grosse. 2020. https://openreview.net/forum?id=SkgsACVKPH Picking winning tickets before training by preserving gradient flow . In International Conference on Learning Representations

2020
[42]

Jiahui Xu, Lu Sun, and Dengji Zhao. 2024. https://doi.org/10.1145/3626772.3657922 MoME : Mixture-of-masked-experts for efficient multi-task recommendation . In SIGIR, pages 2527--2531

2024
[43]

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

2024 arXiv
[44]

Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. https://arxiv.org/abs/2309.12284 Metamath: Bootstrap your own mathematical questions for large language models . Preprint, arXiv:2309.12284

2024 arXiv
[45]

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

2022 arXiv
[46]

Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658

2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

https://www.amd.com/en/products/accelerators/instinct/mi300.html

AMD I nstinct MI 300 S eries A ccelerators. https://www.amd.com/en/products/accelerators/instinct/mi300.html. Accessed: 2024-03-03

work page 2024

[4] [4]

Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vuli \'c . 2021. Composable sparse fine-tuning for cross-lingual transfer. arXiv preprint arXiv:2110.07560

arXiv 2021

[5] [5]

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

arXiv 2021

[6] [6]

Yoshua Bengio, Nicholas L \'e onard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432

arXiv 2013

[7] [7]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

2020

[8] [8]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

arXiv 2021