Pith. sign in

REVIEW 4 major objections 4 minor 38 references

Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes Progtuning, a schedule that progressively freezes low Transformer blocks during fine-tuning, cutting updated parameters by about 25% while keeping or slightly improving GLUE and SQuAD scores.

desk verdict The full-fine-tuning 25% reduction is arithmetic and the accuracy results are competitive, but the PEFT savings in Table 3 are an artifact of Eq. (2) silently dropping transformer blocks whenever T does not divide L. read the letter →

arxiv 2506.21119 v1 pith:OV3TGDOJ submitted 2025-06-26 cs.CL cs.AI

Xiaoshuang Ji , Zhendong Zhao , Xiaojun Chen , Xin Zhao , Zeyao Liu This is my paper · ORCID
classification cs.CLcs.AI
keywords progressivefine-tuningparameter-efficientTransformerblockslayerfreezingGLUEbenchmarkSQuADBERTLoRA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Progtuning is a training schedule for fine-tuning Transformer language models. Instead of updating every parameter in every epoch, it divides the model's Transformer blocks into as many parts as there are training epochs, then trains a shrinking set of blocks: epoch one updates the whole model, and each later epoch freezes one more low-level part until only the highest blocks and the classification head remain trainable. The paper's central claim is that this progressive freezing reduces the total number of parameter updates by roughly 25% on GLUE and SQuAD compared with ordinary fine-tuning, while keeping task accuracy level or slightly better. The schedule also layers onto parameter-efficient methods such as Adapter, BitFit, and LoRA, reducing their update counts further. A sympathetic reader would care because the method changes only the training procedure, not the model, and so offers a nearly free way to lower fine-tuning compute.

What carries the argument

The central mechanism is the shrinking stage schedule. The model is partitioned into $T$ parts of consecutive Transformer blocks, and the trainable set in epoch $t$ is the suffix $S_t = P_t \circ P_{t+1} \circ \cdots \circ P_T \circ H$, so the number of blocks updated decreases monotonically while the blocks nearest the output head stay trainable longest. The same suffix partition is applied not to whole blocks but to the trainable parameters introduced by a parameter-efficient fine-tuning method, such as adapter modules, bias terms, or low-rank matrices, which lets the savings stack on top of parameter-efficient fine-tuning.

What would settle it

Re-run the Adapter+Progtuning GLUE recipe on BERT-base with 12 blocks and 8 epochs while explicitly listing which block belongs to each of the 8 stages; if four blocks never appear in any stage, recompute the 67% updated-parameter saving and the final accuracy using a schedule that covers all 12 blocks. If the corrected saving largely disappears or the accuracy gap widens materially, the central parameter-efficient fine-tuning claim fails.

Watch

Extended reading notes

Core claim

The paper's central discovery is that Transformer blocks contribute unequally to downstream performance, with higher blocks mattering more, and that this asymmetry can be exploited by a time-varying training schedule. Progtuning writes the model as $M = E \circ P_1 \circ \cdots \circ P_T \circ H$, where $T$ is the number of fine-tuning epochs and each part $P_t$ contains $\lfloor L/T \rfloor$ consecutive blocks, then forms overlapping stages $S_t = P_t \circ \cdots \circ P_T \circ H$. In epoch $t$ only stage $S_t$ and the embedding layers are updated, so the highest blocks, closest to the classifier head, remain trainable in every epoch while lower blocks are frozen progressively. Measured on GLUE, the schedule reduces updated parameters by about 25% (30% for BERT-large) and on SQuAD v1.1 and v2.0 by about 26% and 20%, with average scores equal to or slightly above full fine-tuning. On top of Adapter, BitFit, and LoRA, the same partition cuts their updated parameters further, with Adapter dropping by 67%.

Load-bearing premise

The load-bearing premise is that the model's blocks can be split into exactly one equal-sized part per training epoch, with every block assigned to some part; when the number of blocks is not divisible by the number of epochs, the dividing-up as written leaves some blocks out of the schedule entirely.

Editorial extensions

If this is right

  • Fine-tuning can be made cheaper without changing the model architecture, loss function, or pretrained weights; only the set of blocks receiving gradients changes from epoch to epoch.
  • The savings grow with the number of training epochs in the tested range, since more epochs mean more frozen low-level parts and a larger fraction of the schedule that skips them.
  • The benefit is orthogonal to parameter-efficient fine-tuning: methods that already train only a small subset of parameters can still reduce their updated-parameter count further under the same progressive schedule.
  • Because the schedule updates high blocks more often, it behaves as a mild regularizer, which the authors credit for the small average score improvements over full fine-tuning.
  • The ablations imply that low blocks cannot simply be dropped: training them at the start of the schedule is necessary, and reversing the order to progressively grow the model hurts performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the 25% figure is tied to the 2-3 epoch fine-tuning regime used on GLUE and SQuAD; at higher epoch counts the savings could be larger, but the equal-partition rule would need to assign every block explicitly when the number of blocks is not divisible by the number of epochs.
  • Inference: the authors do not attempt to learn the freezing order, so a natural testable extension is to score each block's contribution dynamically and order the freeze schedule by measured importance rather than by fixed depth.
  • Inference: the same suffix-shrinking idea transfers in principle to any stacked network, such as deep convolutional models or stacked generative stages, where later layers are believed to carry task-specific information; a quick test would apply the schedule to a non-Transformer vision model.
  • Inference: the paper reports updated-parameter counts rather than wall-clock time, so a direct measurement of GPU hours and energy per epoch would show whether the 25% update reduction translates into proportional end-to-end savings in practice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes Progtuning, a progressive fine-tuning framework for Transformer language models. The method partitions the L Transformer blocks into T parts, one part per training epoch, and then fine-tunes a shrinking suffix of parts in each epoch, so that later epochs update fewer blocks. The authors report roughly 20-30% reductions in updated parameters on GLUE and SQuAD with competitive accuracy, and they additionally combine Progtuning with Adapter, BitFit, and LoRA, claiming that these PEFT methods also obtain large savings in updated parameters. The central methodological issue is that the partition in Eq. (2) is only a valid decomposition of the model when T divides L, which fails for the PEFT configurations used in the experiments.

Significance. If the claims held, Progtuning would be a simple and attractive way to reduce the backward-pass cost of fine-tuning and of PEFT methods, and the full-fine-tuning ~25% reduction is indeed an arithmetic consequence of the stage definitions and not circular. The reported full fine-tuning results are broadly competitive, and the paper honestly states that training time was not measured. However, the PEFT half of the central claim is built on an invalid decomposition for the configurations used, and the headline Adapter saving is numerically impossible under any valid nested schedule that covers all blocks. Because the PEFT adaptability is advertised as a core contribution, the current results do not support the paper's main claims.

major comments (4)
  1. [§3, Eq. (2) and Algorithm 1] The decomposition M = E∘P_1∘...∘P_T∘H is valid only when T divides L, because each P_t contains floor(L/T) blocks and the union then contains T·floor(L/T) blocks. For BERT-base (L=12) and the PEFT experiments, Section 4.3 states that the number of epochs is "usually 8 or larger"; for T=8,...,11, the right-hand side of Eq. (2) contains only T blocks and silently omits blocks B_{T+1},...,B_{12}, and for T>12 it degenerates. Algorithm 1 line 5 invokes Eq. (2) before building stages, so Tables 3 and 4 are produced by a schedule that never updates several Transformer blocks. The method must specify a partition that covers all blocks, and the experiments must be rerun under that corrected partition.
  2. [Table 3, Adapter+Progtuning row] The reported 67% reduction for Adapter tuning (35.8M to 11.9M) cannot be achieved by any valid progressive schedule with eight nonempty stages covering all 12 blocks. With 8 stages, the minimum cumulative number of block updates is 46 out of 96 block-epochs, giving at most about a 52% saving; including the always-updated embeddings and task head only lowers the possible saving. The 67% figure therefore reflects omitted layers rather than progressive shrinkage, and it invalidates the PEFT claim made in the abstract and conclusion.
  3. [§4.1 and Tables 1, 2, 4] No variance or significance information accompanies the five-run averages, although several decisive comparisons are small (e.g., BERT-base GLUE average of 82.8 vs. 82.6 in Table 1; LoRA+Progtuning 85.8 vs. 86.0 in Table 4). The claims that Progtuning "enhances" performance and that the PEFT trade-offs are "acceptable" are therefore not statistically supported. Please report standard deviations or per-run results and indicate which differences are reliable.
  4. [§4.3, first paragraph] The exact epoch counts T used for Adapter, BitFit, and LoRA on each GLUE task are not reported; the text only says that hyperparameters follow "the settings in original papers" and that T is "usually 8 or larger". Because the number of updated parameters and the validity of Eq. (2) both depend directly on T, this omission prevents reproduction and obscures whether the invalid partition affects some or all PEFT results.
minor comments (4)
  1. [§1, contribution bullet; footnote 4] The contribution bullet states that "Progtuning also reduces training time greatly," but footnote 4 says training time was not included in the experiments; this should be removed or explicitly qualified.
  2. [Eqs. (1)-(5)] The equations use summation symbols where function composition is intended; they should use composition notation, e.g., M = E ∘ B_1 ∘ ... ∘ B_L ∘ H.
  3. [Table 3] The table should define precisely what "updated parameters" means for PEFT methods, including whether the numbers are cumulative over all epochs and how per-task differences arise; otherwise the reported savings are difficult to interpret.
  4. [Figure 1 and Section 5] Figure 1 should state how the per-block performance was measured (which blocks were updated, on which data split, and with what protocol), and the conclusion's attribution of gains to reduced overfitting is post hoc without an experiment that isolates overfitting.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the ~25% parameter-reduction claim is an arithmetic consequence of the schedule, and the performance comparisons are external empirical results.

full rationale

The paper's central resource claim is not a fitted prediction: Eq. (3) defines stages St := Pt ∘ ... ∘ PT ∘ H, and Algorithm 1 updates only the i-th stage in the i-th epoch, so the reduced total of updated parameters is a direct arithmetic consequence of the schedule, not a quantity obtained by fitting. The performance claims in Tables 1, 2, and 4 are external comparisons against standard fine-tuning and PEFT baselines, so they do not reduce to the method's own definitions. There are no load-bearing self-citations: the cited progressive-learning work [10], the BERT layer analysis [4], and the HuggingFace implementation [14] are external priors used for motivation or infrastructure. The 'mitigating overfitting' explanation is post hoc and speculative, but that is an explanatory overclaim, not a circularity. A serious non-circularity issue is that Eq. (2) partitions L=12 blocks into T parts of floor(L/T) blocks each; for the PEFT runs with T>=8, four blocks are silently omitted, so the Table 3 savings (e.g., Adapter 35.8M -> 11.9M) rest on an invalid decomposition. This is a correctness/validity bug, not a circular derivation, and it should be addressed separately.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The ledger is small because the method is a training schedule. The main debt is the unstated divisibility assumption in Eq. (2), the domain assumption about layer contribution, and the hand-chosen stage count T. There are no invented physical entities.

free parameters (2)
  • stage count T = number of training epochs = 3 (GLUE, SQuAD v1.1), 2 (SQuAD v2.0), 8 or larger (PEFT settings)
    T defines the number of parts and stages, and the savings factor (T+1)/(2T) depends directly on it. It is chosen by hand from task conventions, not derived.
  • block partition size floor(L/T) = 4 blocks (T=3), 6 blocks (T=2), 1 block (T=8)
    The paper partitions blocks into parts of size floor(L/T). When L=12 and T=8, this leaves 4 blocks unassigned, silently changing the model being tuned.
assumptions (3)
  • ad hoc to paper Transformer blocks can be partitioned into T contiguous groups of equal size floor(L/T) while preserving the model composition in Eq. (2).
    Eq. (2) writes M as a composition of T parts, but for L=12, T=8, floor(12/8)=1 so only 8 of 12 blocks appear; no remainder handling is specified.
  • domain assumption Low blocks extract low-level information and high blocks dominate task-relevant performance, so training high blocks longer than low blocks is beneficial.
    Motivated by Fig. 1 and Jawahar et al. [4], but not established on the evaluated tasks with error bars or a controlled experiment.
  • domain assumption Frozen early layers can be safely left at epoch-1 weights while later layers continue training; the optimizer does not need to co-adapt them.
    Core to the schedule; supported only by empirical results, with no convergence argument or sensitivity analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models." pith.science (2026). https://pith.science/paper/OV3TGDOJ

@misc{pith2026250621119,
  author       = {Pith},
  title        = {Pith review of: Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OV3TGDOJ}},
  note         = {Machine review of arXiv:2506.21119}
}
read the original abstract

Fine-tuning is a promising technique for leveraging Transformer-based language models in downstream tasks. As model sizes continue to grow, updating all model parameters becomes increasingly costly. Parameter-efficient fine-tuning methods effectively address this issue by selectively updating a small subset of parameters. However, fine-tuning and most existing parameter-efficient fine-tuning methods require updating the same number of parameters as the initial size, ignoring the unequal contribution across Transformer blocks and leading to extremely inefficient allocation of computing resources. In this paper, we propose Progtuning, the novel fine-tuning framework combined with progressive learning for Transformer-based language models. Specifically, Progtuning progressively reduces the number of updated transformer blocks based on the contribution. Remarkably, Progtuning optimizes resource allocation and reduces the number of updated parameters by approximately 25\%, while still maintaining competitive performance. And it also exhibits high adaptability with parameter-efficient fine-tuning methods, demonstrating excellent performance across various adaptation scenarios.

Figures

Figures reproduced from arXiv: 2506.21119 by the authors.

Figure 1
Figure 1. The performance of Transformer blocks. We only update embedding layers and selected Transformer block in fine-tuning. The model we use is BERTBASE and trained on SST-2 dataset. However, fine-tuning and most existing PEFT methods update all trainable parameters as the initial size during the whole training process, ignoring the unequal contribution to model performance across Transformer blocks. And this results in n… view at source ↗
Figure 2
Figure 2. The structure of a Transformer-based language model. 2.2 Fine-tuning Fine-tuning is the predominant model training technique used in the realm of NLP [16,17]. Pretrained models are trained on specific datasets to acquire domain-specific knowledge of downstream tasks. Fine-tuning modifies all pa￾rameters and saves a full copy of pretrained parameters for each task. Since the model has been pretrained on the general c… view at source ↗
Figure 3
Figure 3. The overview of Progtuning. We divide the model into T parts and arrange them to form overlapping and shrinking stages. Then we arrange these parts to construct T overlapping and shrinking stages, denoted by St, for t ∈ {1, ..., T}. Overlapping means Transformer blocks are shared by all stages. Shrinking means the number of Transformer blocks is de￾creasing. The stage St is the combination of the parts numbered from… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Updated parameters on GLUE. FT stands for fine-tuning and PT stands for Progtuning [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Progtuning without low blocks. of every stage during training. In other words, the first part will not be trained and the last stage only consists of the header H as shown in [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Progtuning that the stages are progressively growing. keep growing to contain more low parts, which can be defined: St := X T i=T−t+1 Pi ◦ H = PT−t+1 ◦ PT−t+2 ◦ · · · ◦ PT ◦ H. (5) The training procedure starts from the first stage and progressively extends to the full…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 14 canonical work pages

  1. [17]

    Universal language model fine-tuning for text classifica- tion,

    J. Howard and S. Ruder, “Universal language model fine-tuning for text classifica- tion,”arXiv preprint arXiv:1801.06146, 2018

    arXiv 2018

  2. [1]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  3. [2]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,”arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  4. [3]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Nee- lakantan, P. Shyam, G. Sastry, A. Askellet al., “Language models are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877– 1901, 2020

    work page 1901

  5. [4]

    What does bert learn about the structure of language?

    G. Jawahar, B. Sagot, and D. Seddah, “What does bert learn about the structure of language?” inACL 2019-57th Annual Meeting of the Association for Computa- tional Linguistics, 2019

    work page 2019

  6. [5]

    Parameter-efficient transfer learning for nlp,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Ges- mundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” inInternational Conference on Machine Learning. PMLR, 2019, pp. 2790–2799

    work page 2019

  7. [6]

    Prefix-tuning: Optimizing continuous prompts for genera- tion,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for genera- tion,”arXiv preprint arXiv:2101.00190, 2021

    arXiv 2021

  8. [7]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,”arXiv preprint arXiv:2106.09685, 2021

    arXiv 2021

Show all 38 references
  1. [8]

    A fully progressive approach to single-image super-resolution,

    Y. Wang, F. Perazzi, B. McWilliams, A. Sorkine-Hornung, O. Sorkine-Hornung, and C. Schroers, “A fully progressive approach to single-image super-resolution,” inProceedings of the IEEE conference on computer vision and pattern recognition workshops, 2018, pp. 864–873

    2018

  2. [9]

    Cascade ef-gan: Progressive facial expres- sion editing with local focuses,

    R. Wu, G. Zhang, S. Lu, and T. Chen, “Cascade ef-gan: Progressive facial expres- sion editing with local focuses,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 5021–5030

    2020

  3. [10]

    Progressive growing of gans for improved quality, stability, and variation,

    T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,”arXiv preprint arXiv:1710.10196, 2017

    2017 arXiv

  4. [11]

    Glue: A multi-task benchmark and analysis platform for natural language understanding,

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  5. [12]

    Squad: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ questions for machine comprehension of text,”arXiv preprint arXiv:1606.05250, 2016. Progtuning: Progressive Fine-tuning Framework for Transformer-based 13

    2016 arXiv

  6. [13]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models,

    E. B. Zaken, S. Ravfogel, and Y. Goldberg, “Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models,”arXiv preprint arXiv:2106.10199, 2021

    2021

  7. [14]

    Transformers: State-of-the-art natural language processing,

    T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowiczet al., “Transformers: State-of-the-art natural language processing,” inProceedings of the 2020 conference on empirical methods in natural language processing: system demons...

    2020

  8. [15]

    Parameter-efficient transfer learning with diff pruning,

    D. Guo, A. M. Rush, and Y. Kim, “Parameter-efficient transfer learning with diff pruning,”arXiv preprint arXiv:2012.07463, 2020

    2012 arXiv

  9. [16]

    Semi-supervised sequence learning,

    A. M. Dai and Q. V. Le, “Semi-supervised sequence learning,”Advances in neural information processing systems, vol. 28, 2015

    2015

  10. [18]

    Scaling down to scale up: A guide to parameter-efficient fine-tuning,

    V. Lialin, V. Deshpande, and A. Rumshisky, “Scaling down to scale up: A guide to parameter-efficient fine-tuning,”arXiv preprint arXiv:2303.15647, 2023

    2023 arXiv

  11. [19]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,”arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  12. [20]

    Huggingface’s transformers: State-of- the-art natural language processing,

    T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowiczet al., “Huggingface’s transformers: State-of- the-art natural language processing,”arXiv preprint arXiv:1910.03771, 2019

    1910 arXiv

  13. [21]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi` ere, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  14. [22]

    Exploring the limits of transfer learning with a unified text-to- text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to- text transformer,”The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485–5551, 2020

    2020

  15. [23]

    Loftq: Lora-fine-tuning-aware quantization for large language models,

    Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao, “Loftq: Lora-fine-tuning-aware quantization for large language models,”arXiv preprint arXiv:2310.08659, 2023

    2023 arXiv

  16. [24]

    P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks,

    X. Liu, K. Ji, Y. Fu, W. L. Tam, Z. Du, Z. Yang, and J. Tang, “P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks,”arXiv preprint arXiv:2110.07602, 2021

    2021 arXiv

  17. [25]

    Qlora: Efficient fine- tuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient fine- tuning of quantized llms,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  18. [26]

    Efficient fine-tuning of bert models on the edge,

    D. Vucetic, M. Tayaranian, M. Ziaeefard, J. J. Clark, B. H. Meyer, and W. J. Gross, “Efficient fine-tuning of bert models on the edge,” in2022 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 2022, pp. 1838–1842

    2022

  19. [27]

    Krona: Parameter efficient tuning with kronecker adapter,

    A. Edalati, M. Tahaei, I. Kobyzev, V. P. Nia, J. J. Clark, and M. Reza- gholizadeh, “Krona: Parameter efficient tuning with kronecker adapter,”arXiv preprint arXiv:2212.10650, 2022

    2022 arXiv

  20. [28]

    Roberta: A robustly optimized bert pretraining ap- proach,

    Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettle- moyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining ap- proach,”arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  21. [29]

    Gpt under- stands, too,

    X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang, “Gpt under- stands, too,”AI Open, 2023

    2023

  22. [30]

    Dora: Enhancing parameter-efficient fine-tuning with dynamic rank distribution,

    Y. Mao, K. Huang, C. Guan, G. Bao, F. Mo, and J. Xu, “Dora: Enhancing parameter-efficient fine-tuning with dynamic rank distribution,”arXiv preprint arXiv:2405.17357, 2024. 14 X. Ji et al

    2024 arXiv

  23. [31]

    Pre-trained models for natural language processing: A survey,

    X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang, “Pre-trained models for natural language processing: A survey,”Science China technological sciences, vol. 63, no. 10, pp. 1872–1897, 2020

    2020

  24. [32]

    Sparsegpt: Massive language models can be accurately pruned in one-shot,

    E. Frantar and D. Alistarh, “Sparsegpt: Massive language models can be accurately pruned in one-shot,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 10 323–10 337

    2023

  25. [33]

    A simple and effective pruning approach for large language models,

    M. Sun, Z. Liu, A. Bair, and J. Z. Kolter, “A simple and effective pruning approach for large language models,”arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  26. [34]

    Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale,

    T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale,”Advances in Neural Information Processing Systems, vol. 35, pp. 30 318–30 332, 2022

    2022

  27. [35]

    Llm-fp4: 4-bit floating- point quantized transformers,

    S.-y. Liu, Z. Liu, X. Huang, P. Dong, and K.-T. Cheng, “Llm-fp4: 4-bit floating- point quantized transformers,”arXiv preprint arXiv:2310.16836, 2023

    2023 arXiv

  28. [36]

    Parameter-efficient fine-tuning for large models: A comprehensive survey,

    Z. Han, C. Gao, J. Liu, S. Q. Zhanget al., “Parameter-efficient fine-tuning for large models: A comprehensive survey,”arXiv preprint arXiv:2403.14608, 2024

    2024 arXiv

  29. [37]

    Deep residual learning for image recog- nition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recog- nition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  30. [38]

    Layer normalization,

    J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,”arXiv preprint arXiv:1607.06450, 2016

    2016 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.