Pith. sign in

REVIEW 5 major objections 5 minor 75 references

Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Parameter-efficient fine-tuning is a strict subset of full fine-tuning, with smaller representational capacity and weaker robustness.

desk verdict Broad empirical comparison, but the central theory is unproven: Theorem 1's strict-subset claim is a non-sequitur and Theorems 4-5 have algebraic errors. read the letter →

arxiv 2505.22355 v1 pith:ZKTBF3HT submitted 2025-05-28 cs.LG

Yongkang Liu , Xingle Xu , Ercong Nie , Zijing Wang , Shi Feng , Daling Wang , Qian Li , Hinrich Schütze This is my paper
classification cs.LG MSC 68T0768Q32
keywords parameter-efficientfine-tuningfullrepresentationalcapacityrobustnessLoRAlow-rankadaptationRademachercomplexityadversarial
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that parameter-efficient fine-tuning (PEFT), which updates only a small set of weights, is fundamentally more limited than full fine-tuning (FFT), which updates every weight. It claims PEFT is a strict subset of FFT: any model reachable by PEFT is also reachable by FFT, but most FFT outcomes are not reachable by PEFT. The authors prove upper bounds showing that PEFT's small parameter space caps how much a model can adapt, makes it more sensitive to input perturbations, and reduces the benefit it gets from additional training data. On 15 datasets and 11 adversarial test sets, FFT generally wins on complex reasoning, generation, and instruction-following tasks, while PEFT can match or beat FFT on simpler benchmarks and few-shot settings.

What carries the argument

The load-bearing object is the reparameterization map $g: \mathbb{R}^k \to \mathbb{R}^d$ of Equation 4, which sends the low-dimensional PEFT parameters $\Phi$ into the full parameter space. Because $k \ll d$, its image has Lebesgue measure zero in $\mathbb{R}^d$ (via Sard's theorem), and the paper uses this to conclude that the PEFT function class sits strictly inside the FFT function class. The supporting machinery consists of a layer-wise norm-and-Lipschitz bound that yields the capacity upper bound, rank-truncated singular value decomposition with exponentially decaying singular values that yields the critical-rank result, a projected-Hessian expansion that separates perturbations into components inside and orthogonal to the PEFT subspace, and Rademacher complexity bounds that yield the $\sqrt{k/N}$ versus $\sqrt{d/N}$ generalization rates behind the marginal-benefit statement.

What would settle it

Train a LoRA and an FFT model on the same reasoning task to convergence, then search for a full fine-tuned parameter vector that realizes exactly the same input-output mapping as the LoRA model while lying outside the LoRA subspace; if one exists, the function-level strict-subset claim fails. Alternatively, measure the per-sample risk reduction of each method on a large-scale training set: Theorem 5 predicts the ratio of PEFT to FFT marginal gains approaches $O(k/d)$, so observing a ratio far larger than $k/d$ would falsify it. And on an adversarial test set, equal robustness between LoRA and FFT after matched training would contradict Theorem 4.

Watch

Extended reading notes

Core claim

The central claim is that PEFT is a strict subset of FFT, both in the space of reachable parameters and in the set of functions the model can represent. The paper formalizes PEFT as $f(x; \theta_0 + g(\Phi))$ with $g: \mathbb{R}^k \to \mathbb{R}^d$ non-surjective and $k \ll d$, so the reachable parameters form a $k$-dimensional, measure-zero submanifold of the full parameter space. From this starting point it proves that PEFT has a capacity upper bound determined by the number of tunable parameters, that beyond a critical rank added parameters give diminishing returns, that PEFT loses more than FFT under the same perturbation, and that each extra training sample reduces PEFT's expected risk by a factor of $O(k/d)$ relative to FFT. The experiments are presented as validation: FFT outperforms PEFT on math reasoning, SQL generation, instruction fine-tuning, and most adversarial test sets, while PEFT shows competitive or superior results on simple classification tasks and in low-data regimes.

Load-bearing premise

The paper assumes that reducing the number of trainable parameters genuinely reduces the set of behaviors a model can produce, which requires different parameter settings to yield different behaviors; overparameterized networks can produce the same behavior from many different parameter settings, so this premise, used at Equation 4 and Appendix F.1, is not automatic.

Editorial extensions

If this is right

  • Any task PEFT can solve, FFT can also solve, so FFT is the safer choice whenever compute and memory allow.
  • The size of the tunable parameter budget directly caps how much the model can adapt; complex tasks that require new features should show a persistent FFT advantage.
  • Adding more data helps FFT more than PEFT, so the performance gap should widen as training sets grow and narrow in few-shot regimes.
  • Increasing LoRA rank beyond a critical value will not close the gap to FFT; the model's expressive power saturates.
  • Adversarial or out-of-distribution evaluation should expose larger FFT advantages because PEFT cannot cancel perturbation components orthogonal to its parameter subspace.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the strict-subset claim is interpreted at the level of functions rather than parameters, it silently requires the parameter-to-function map to be injective or dimension-preserving; overparameterized networks can realize the same behavior from many parameter settings, so the parameter-space argument does not automatically transfer to function space.
  • The $O(k/d)$ marginal-benefit formula implies a concrete testable curve: the FFT advantage should grow smoothly with data volume and eventually dominate PEFT on large corpora, matching the paper's few-shot observation that PEFT wins when the training set is small.
  • The theory suggests that PEFT variants which adapt the low-dimensional subspace during training, rather than keeping it fixed, could shrink the orthogonal perturbation component and recover some robustness; the paper does not test this class of hybrids.
  • The layer-dependence in the capacity bound, where parameters nearer the output layer matter more, implies that allocating a PEFT budget to later layers should be more effective than uniform allocation; this allocation rule is not directly tested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a theoretical comparison between parameter-efficient fine-tuning (PEFT) and full fine-tuning (FFT). Its central claims are that PEFT is a strict subset of FFT (Theorem 1), that PEFT has a bounded representational capacity that limits adaptation (Theorem 2), that there are diminishing marginal benefits to increasing PEFT parameters (Theorem 3), that PEFT is more sensitive to perturbations than FFT (Theorem 4), and that PEFT's marginal benefit from additional training samples is only O(k/d) that of FFT (Theorem 5). These claims are supported by extensive experiments on 15 datasets and 11 adversarial test sets, with code released. The empirical study is broad, but the theoretical results are the load-bearing part of the paper, and several of the proofs are invalid.

Significance. If the theorems were correct, they would give a general theoretical argument that PEFT is inherently weaker than FFT in representational capacity, robustness, and data efficiency. The empirical contribution is substantial in scope, including comparisons across multiple model families and adversarial benchmarks, and the finding that FFT tends to outperform PEFT on complex and adversarial tasks is a useful datapoint for practitioners. However, the theoretical apparatus is not merely an auxiliary motivation: it is the paper's stated contribution. Since the main theorems contain serious logical and algebraic errors, the central claims are not established, and the paper cannot be accepted in its current form.

major comments (5)
  1. [Section 3, Theorem 1; Appendix F.1]
  2. [Section 3, Theorem 2; Eq. (6); Appendix F.2]
  3. [Section 3, Theorem 3; Eq. (7); Appendix F.3]
  4. [Section 3, Theorem 4; Eq. (8); Appendix F.4]
  5. [Section 3, Theorem 5; Eqs. (12)-(19)]
minor comments (5)
  1. [Section 2, Eq. (3)]
  2. [Section 3, Theorem 1; Eq. (5)]
  3. [Section 3, Theorem 4]
  4. [Appendix F.4, Eq. (53)]
  5. [Tables and figures]

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: central derivations are self-contained; Theorem 1's flaw is invalidity, not circularity.

full rationale

No fitted constants are renamed as predictions and the empirical benchmarks are external, so the paper is not circular in the fitted-input sense. Theorems 1–5 are derived from stated assumptions: Sard's theorem for the parameter map, Lipschitz norm bounds for the capacity bound, Eckart–Young–Mirsky plus an external singular-value-decay assumption for diminishing returns, Rademacher complexity for the sample-size bound, and a Taylor expansion for the robustness comparison. The inclusion half of Theorem 1 is immediate from Eq. (4) by setting theta_Phi = theta0 + g(Phi) in R^d; the paper's additional strictness claim rests on non-surjectivity of the parameter map g, which is a logical error (measure-zero parameter image does not imply measure-zero function class) rather than a circular reduction. There is no self-citation chain used to force the central claim: reference [42] is an overlapping-author citation but is used only as a general pointer to FFT performance, not as the load-bearing justification for any theorem. The Appendix E limitation statement explicitly acknowledges the lack of a strict task-complexity definition, which is a transparency note rather than a circular step. Overall, the derivations do not assume their conclusions; they are best assessed on correctness grounds, not circularity grounds.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The derivations import several unstated assumptions; the most damaging is the measure-zero-to-function-class implication, which is not true. Exponential singular value decay, positivity and eigenvalue conditions on the Hessian, additive weight-perturbation structure, and parameter-count Rademacher bounds are also assumed rather than established. No free parameters are fitted to data.

assumptions (6)
  • ad hoc to paper Non-surjective parameter map implies strictly smaller function class.
    Used to prove Theorem 1; false when parameter-to-function map is non-injective, which is typical for overparameterized networks.
  • domain assumption The ideal weight update has exponentially decaying singular values, sigma_i approximately O(e^{-lambda i}).
    Invoked in Theorem 3 (Appendix F.3) to define a critical rank r_c; cited to [13], but the transfer to LoRA or prefix updates is not demonstrated.
  • domain assumption Hessian H is positive definite and satisfies H^3 is greater than H in the Loewner order.
    Theorem 4's final inequality requires eigenvalue assumptions beyond positive definiteness; positive definite alone allows eigenvalues below 1 where H^3 is not larger than H.
  • domain assumption All PEFT methods can be written as f(x; theta0 + g(Phi)) with a linear map g = P Phi.
    Equation 4 and Theorem 4 assume additive weight perturbation; prefix tuning and adapters modify activations or add parameters outside the weight matrices, so the projection model does not cover them.
  • domain assumption Rademacher complexity of the fine-tuning class is O(sqrt(d/N)) with d equal to parameter count.
    Theorem 5 relies on this parameter-count bound; without norm or geometry restrictions such bounds are not generally valid for deep networks.
  • domain assumption The fine-tuning model has reached the empirical optimum, L_N(theta_hat_N) - L*_H tends to 0.
    Explicitly assumed before Eq. 11 to support the risk decomposition in Theorem 5; not guaranteed in practice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning." pith.science (2026). https://pith.science/paper/ZKTBF3HT

@misc{pith2026250522355,
  author       = {Pith},
  title        = {Pith review of: Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZKTBF3HT}},
  note         = {Machine review of arXiv:2505.22355}
}
read the original abstract

Parameter-Efficient Fine-Tuning (PEFT) methods achieve performance comparable to Full Fine-Tuning (FFT) while requiring significantly fewer computing resources, making it the go-to choice for researchers. We find that although PEFT can achieve competitive results on some benchmarks, its performance falls short of FFT in complex tasks, such as reasoning and instruction-based fine-tuning. In this paper, we compare the characteristics of PEFT and FFT in terms of representational capacity and robustness based on optimization theory. We theoretically demonstrate that PEFT is a strict subset of FFT. By providing theoretical upper bounds for PEFT, we show that the limited parameter space constrains the model's representational ability, making it more susceptible to perturbations. Experiments on 15 datasets encompassing classification, generation, reasoning, instruction fine-tuning tasks and 11 adversarial test sets validate our theories. We hope that these results spark further research beyond the realms of well established PEFT. The source code is in the anonymous Github repository\footnote{https://github.com/misonsky/PEFTEval}.

Figures

Figures reproduced from arXiv: 2505.22355 by the authors.

Figure 1
Figure 1. Incremental parameter distributions for different fine-tuning methods. For Prefix, the [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. The impact of training samples on the perfor [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Performance trends of FFT and LoRA fine-tuning with different amounts of fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

75 extracted references · 38 canonical work pages

  1. [1]

    Composable sparse fine-tuning for cross-lingual transfer

    Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vuli´c. Composable sparse fine-tuning for cross-lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1778–1796, 2022

    work page 2022

  2. [2]

    Fine-Tuning LLMs: LoRA or Full-Parameter? An in-depth Analysis with Llama-2

    Niederfahrenhorst Artur, Hakhamaneshi Kourosh, and Ahmad Rehaan. Fine-Tuning LLMs: LoRA or Full-Parameter? An in-depth Analysis with Llama-2 . 2023. URL https://www. anyscale.com/blog/fine-tuning-llms-lora-or-full-parameter

    work page 2023

  3. [3]

    Machine learning theory

    Avrim Blum. Machine learning theory. Carnegie Melon Universit, School of Computer Science, 26:5, 2007

    work page 2007

  4. [4]

    Attention fusion: a light yet efficient late fusion mechanism for task adaptation in nlu

    Jin Cao, Chandana Satya Prakash, and Wael Hamza. Attention fusion: a light yet efficient late fusion mechanism for task adaptation in nlu. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 857–866, 2022

    work page 2022

  5. [5]

    SemEval- 2019 task 3: EmoContext contextual emotion detection in text

    Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi, and Puneet Agrawal. SemEval- 2019 task 3: EmoContext contextual emotion detection in text. In Jonathan May, Ekaterina Shutova, Aurelie Herbelot, Xiaodan Zhu, Marianna Apidianaki, and Saif M. Mohammad, editors, Proceedings of the 13th International Workshop on Semantic Evaluation, pages 39–48, Minneap...

    doi:10.18653/v1/s19-2005 2019

  6. [6]

    Parameter-efficient fine-tuning design spaces

    Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang. Parameter-efficient fine-tuning design spaces. arXiv preprint arXiv:2301.01821, 2023

    arXiv 2023

  7. [7]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/

    2023

  8. [8]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

Show all 75 references
  1. [9]

    A farewell to the bias-variance tradeoff? an overview of the theory of overparameterized machine learning

    Yehuda Dar, Vidya Muthukumar, and Richard G Baraniuk. A farewell to the bias-variance tradeoff? an overview of the theory of overparameterized machine learning. arXiv preprint arXiv:2109.02355, 2021

    2021 arXiv

  2. [10]

    Identifying and attacking the saddle point problem in high-dimensional non- convex optimization

    Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non- convex optimization. Advances in neural information processing systems, 27, 2014

    2014

  3. [11]

    Parameter- efficient fine-tuning of large-scale pre-trained language models

    Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. Parameter- efficient fine-t...

    2023

  4. [12]

    Krona: Parameter efficient tuning with kronecker adapter

    Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212.10650, 2022

    2022 arXiv

  5. [13]

    Jordan, and Zhengjun Zha

    Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael I. Jordan, and Zhengjun Zha. Rank diminishing in deep neural networks. ArXiv, abs/2206.06072, 2022. URL https: //api.semanticscholar.org/CorpusID:249625522

    2022 arXiv

  6. [14]

    Sharpness-aware min- imization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, volume abs/2010.01412, 2020. URL https://api.semanticscholar. org/CorpusID:222134093. 10

    2010 arXiv

  7. [15]

    Robustness gym: Unifying the NLP evaluation landscape

    Karan Goel, Nazneen Fatema Rajani, Jesse Vig, Zachary Taschdjian, Mohit Bansal, and Christopher Ré. Robustness gym: Unifying the NLP evaluation landscape. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Huma...

    2021

  8. [16]

    A survey of adversarial defenses and robustness in nlp

    Shreya Goyal, Sumanth Doddapaneni, Mitesh M Khapra, and Balaraman Ravindran. A survey of adversarial defenses and robustness in nlp. ACM Computing Surveys, 55(14s):1–39, 2023

    2023

  9. [17]

    Parameter-efficient fine-tuning for large models: A comprehensive survey

    Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608, 2024

    2024 arXiv

  10. [18]

    Parameter efficient instruction tuning: An empirical study

    Pengfei He. Parameter efficient instruction tuning: An empirical study. arXiv preprint arXiv:2411.16775, 2024

    2024 arXiv

  11. [19]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29,

    2022

  12. [20]

    Llm-adapters: An adapter family for parameter-efficient fine- tuning of large language models

    Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm-adapters: An adapter family for parameter-efficient fine- tuning of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natu...

    2023

  13. [21]

    Hira: Parameter-efficient hadamard high-rank adaptation for large language models

    Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. Hira: Parameter-efficient hadamard high-rank adaptation for large language models. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  14. [22]

    Adversarial examples for evaluating reading comprehension systems

    Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 2021–2031. Association for Comp...

    2017

  15. [23]

    Adversarial examples for evaluating reading comprehension systems

    Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328, 2017

    2017 arXiv

  16. [24]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  17. [25]

    Weisen Jiang, Baijiong Lin, Han Shi, Yu Zhang, Zhenguo Li, and James T. Kwok. Effective and parameter-efficient reusing fine-tuned models. CoRR, abs/2310.01886, 2023

    2023 arXiv

  18. [26]

    Viggo: A video game corpus for data-to-text generation in open-domain conversation

    Juraj Juraska, Kevin Bowden, and Marilyn Walker. Viggo: A video game corpus for data-to-text generation in open-domain conversation. In Proceedings of the 12th International Conference on Natural Language Generation, pages 164–172, 2019

    2019

  19. [27]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  20. [28]

    Compacter: Efficient low-rank hypercomplex adapter layers

    Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022–1035, 2021

    2021

  21. [29]

    Bias plus variance decomposition for zero-one loss functions

    Ron Kohavi and David Wolpert. Bias plus variance decomposition for zero-one loss functions. In Proceedings of the Thirteenth International Conference on International Conference on Machine Learning, pages 275–283, 1996

    1996

  22. [30]

    Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Models to Unique Applications

    Hakhamaneshi Kourosh and Ahmad Rehaan. Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Models to Unique Applications . 2023. URL https://www.anyscale. com/blog. 11

    2023

  23. [31]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, 2021

    2021

  24. [32]

    BART: denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Ann...

    2020

  25. [33]

    Measuring the intrinsic dimension of objective landscapes

    Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018

    2018

  26. [34]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pap...

    2021

  27. [35]

    Scaling down to scale up: A guide to parameter-efficient fine-tuning

    Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down to scale up: A guide to parameter-efficient fine-tuning. arXiv preprint arXiv:2303.15647, 2023

    2023 arXiv

  28. [36]

    Mala-500: Massive language adaptation of large language models

    Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André FT Martins, and Hinrich Schütze. Mala-500: Massive language adaptation of large language models. arXiv preprint arXiv:2401.13303, 2024

    2024 arXiv

  29. [37]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Info...

    2022

  30. [38]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965, 2022

    1950

  31. [39]

    A robust adversarial training approach to machine reading comprehension

    Kai Liu, Xin Liu, An Yang, Jing Liu, Jinsong Su, Sujian Li, and Qiaoqiao She. A robust adversarial training approach to machine reading comprehension. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8392–8400, 2020

    2020

  32. [40]

    Roberta: A robustly optimized BERT pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019

    1907 arXiv

  33. [41]

    Multilingual denoising pre-training for neural machine translation

    Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Trans. Assoc. Comput. Linguistics, 8:726–742, 2020

    2020

  34. [42]

    Hift: A hierarchical full parameter fine-tuning strategy

    Yongkang Liu, Yiqun Zhang, Qian Li, Shi Feng, Daling Wang, Yifei Zhang, and Hinrich Schütze. Hift: A hierarchical full parameter fine-tuning strategy. arXiv preprint arXiv:2401.15207, 2024

    2024 arXiv

  35. [43]

    Hidden factors and hidden topics: understanding rating dimensions with review text

    Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pages 165–172, 2013

    2013

  36. [44]

    L. Mirsky. Symmetric gauge functions and unitarily invariant norms. Quarterly Journal of Mathematics, 11:50–59, 1960. URL https://api.semanticscholar.org/CorpusID: 120585992

    1960

  37. [45]

    Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi

    John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Syste...

    2020

  38. [46]

    A modern take on the bias-variance tradeoff in neural networks

    Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste- Julien, and Ioannis Mitliagkas. A modern take on the bias-variance tradeoff in neural networks. arXiv e-prints, pages arXiv–1810, 2018

    2018

  39. [47]

    Learn more, but bother less: parameter efficient continual learning

    Fuli Qiao and Mehrdad Mahdavi. Learn more, but bother less: parameter efficient continual learning. Advances in Neural Information Processing Systems, 37:97476–97498, 2024

    2024

  40. [48]

    Know what you don’t know: Unanswerable ques- tions for squad

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable ques- tions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, 2018

    2018

  41. [49]

    Finetuning LLMs with LoRA and QLoRA: Insights from Hundreds of Experiments

    Sebastian Raschka. Finetuning LLMs with LoRA and QLoRA: Insights from Hundreds of Experiments. 2023. URL https://lightning.ai/pages/community/lora-insights

    2023

  42. [50]

    The measure of the critical values of differentiable maps

    Arthur Sard. The measure of the critical values of differentiable maps. 1942

    1942

  43. [51]

    Recursive deep models for semantic compositionality over a sentiment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language proce...

    2013

  44. [52]

    Stanford alpaca: An instruction-following llama model, 2023

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023

    2023

  45. [53]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  46. [54]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017

    2017

  47. [55]

    Building a question answering test collection

    Ellen M V oorhees and Dawn M Tice. Building a question answering test collection. InProceed- ings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 200–207, 2000

    2000

  48. [56]

    Efficient fine-tuning of bert models on the edge

    Danilo Vucetic, Mohammadreza Tayaranian, Maryam Ziaeefard, James J Clark, Brett H Meyer, and Warren J Gross. Efficient fine-tuning of bert models on the edge. In2022 IEEE International Symposium on Circuits and Systems (ISCAS), pages 1838–1842. IEEE, 2022

    2022

  49. [57]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NL...

    2018

  50. [58]

    Adversarial GLUE: A multi-task benchmark for robustness evaluation of language models

    Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial GLUE: A multi-task benchmark for robustness evaluation of language models. In Proceedings of the Neural Information Processing Systems Track on Datasets and B...

    2021

  51. [59]

    On the robustness of chatgpt: An adversarial and out-of-distribution perspective

    Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, Binxing Jiao, Yue Zhang, and Xing Xie. On the robustness of chatgpt: An adversarial and out-of-distribution perspective. CoRR, abs/2302.12095, 2023

    2023 arXiv

  52. [60]

    The robustness of computer vision models against common corruptions: a survey

    Shunxin Wang, Raymond Veldhuis, and Nicola Strisciuglio. The robustness of computer vision models against common corruptions: a survey. arXiv preprint arXiv:2305.06024, 2023

    2023 arXiv

  53. [61]

    Textflint: Unified multilingual robustness evaluation toolkit for natural language processing

    Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, Qinzhuo Wu, Zhengyan Li, Chong Zhang, Ruotian Ma, Zichu Fei, Ruijian Cai, Jun Zhao, Xingwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Shan Qin...

    2021

  54. [62]

    A broad-coverage challenge corpus for sentence understanding through inference

    Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL-HLT, pages 1112–1122, 2018

    2018

  55. [63]

    Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887, 2018

    2018 arXiv

  56. [64]

    Revisiting out-of-distribution robustness in NLP: benchmark, analysis, and llms evaluations

    Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. Revisiting out-of-distribution robustness in NLP: benchmark, analysis, and llms evaluations. CoRR, abs/2306.04618, 2023

    2023 arXiv

  57. [65]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

    Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–9, 2022

    2022

  58. [66]

    Glm-130b: An open bilingual pre-trained model

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022

    2022 arXiv

  59. [67]

    Openattack: An open-source textual adversarial attack toolkit

    Guoyang Zeng, Fanchao Qi, Qianrui Zhou, Tingji Zhang, Zixian Ma, Bairu Hou, Yuan Zang, Zhiyuan Liu, and Maosong Sun. Openattack: An open-source textual adversarial attack toolkit. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computat...

    2021

  60. [68]

    Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In International Conference on Learning Representations. Openreview, 2023

    2023

  61. [69]

    Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer...

    2022 arXiv

  62. [70]

    Character-level convolutional networks for text classification

    Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015

    2015

  63. [71]

    Towards adaptive prefix tuning for parameter-efficient language model fine-tuning

    Zhenru Zhang, Chuanqi Tan, Haiyang Xu, Chengyu Wang, Jun Huang, and Songfang Huang. Towards adaptive prefix tuning for parameter-efficient language model fine-tuning. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short ...

    2023

  64. [72]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024

    2024

  65. [73]

    Seq2sql: Generating structured queries from natural language using reinforcement learning

    Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103, 2017

    2017 arXiv

  66. [74]

    q_proj",

    Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.04528, 2023. 15 A Ba...

    2023 arXiv

  67. [2022]

    URL https://openreview.net/forum?id=nZeVKeeFYf9

    OpenReview.net, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.