Pith. sign in

REVIEW 3 major objections 6 minor 117 references

Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that decomposing weight matrices along task-sample activation covariances concentrates a task's capability in a few principal components, so freezing those components preserves knowledge and adapting them speeds learning.

desk verdict A useful, well-tested extension of CorDA with two clean new mechanisms; just fix the speedup wording and one mischaracterized baseline. read the letter →

arxiv 2506.13187 v1 pith:T2SUDJCK submitted 2025-06-16 cs.LG cs.AIcs.CLcs.CV

Yibo Yang , Sihao Liu , Chuan Rao , Bang An , Tiancheng Shen , Philip H.S. Torr , Ming-Hsuan Yang , Bernard Ghanem This is my paper
classification cs.LGcs.AIcs.CLcs.CV
keywords low-rankadaptationparameter-efficientfine-tuningcontext-orientedSVDknowledgepreservationcatastrophicforgettingquantizeddynamicrankallocationvision-languagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that low-rank adapters should be initialized from data, not from random noise: by decomposing each weight matrix together with the covariance of its inputs on a few samples of the task that matters, the leading singular directions become the directions that carry that task. If true, one mode can freeze those directions to stop fine-tuning from wiping out pre-trained knowledge, and another can adapt them to learn faster, with quantized fine-tuning reported up to 4.5x faster than QLoRA. The paper also proposes two dynamic strategies, covariance selection and rank allocation, that make the task-specific components more compact and improve both modes. The authors support this by showing that task-sample covariances from related tasks look alike, that discarding the bottom components of their decomposition barely hurts perplexity, and that freezing or adapting those components beats LoRA and several successors.

What carries the argument

The load-bearing object is the context-oriented SVD (CO-SVD). For each linear layer the paper computes $C = XX^T$ from input activations over 256 sampled target-task examples, decomposes $WC = U\Sigma V^T$, and reconstructs $W$ as $U\Sigma(V^T C^{-1})$, with a diagonal-regularization fallback when $C$ is not invertible. Because $C$ encodes the activation outliers a task triggers, the largest singular directions of this decomposition align with the weight directions that carry that task's capability. The paper then uses the bound $\|\Delta Y\|_1 \le \sqrt{d_{\text{out}}}\,\sigma_{\max}(C)/\sigma_{\min}(C)\cdot\sigma_{-r}$ to define a compactness metric, and builds dynamic covariance selection and dynamic rank allocation on it. That metric is what lets the paper choose, per layer, the most representative covariance matrix and the most efficient rank budget.

What would settle it

Run the knowledge-preserved mode using a covariance matrix built from the same 256 samples whose labels are shuffled, or from random noise activations, and measure TriviaQA and NQ-open exact match after fine-tuning on Math; if forgetting is reduced as much as with true task samples, the claim that task-specific covariance orientation is what protects knowledge is not supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that task awareness should enter low-rank adaptation at initialization, through the data covariance, not just through the training loss. Concretely, it claims that decomposing $W C$ rather than $W$ compacts the task's capability into the principal components: freezing those components in knowledge-preserved mode retains the pre-trained knowledge they encode, while adapting them in instruction-previewed mode starts the adapter already aligned with the task and therefore accelerates convergence. On this basis CorDA++ reports better downstream results than LoRA while keeping TriviaQA, NQ open, and WebQS scores far higher after fine-tuning, and in the quantized setting reports up to 4.5x faster convergence than QLoRA. It also claims that the dynamic strategies, selecting the most compact covariance among sampled candidates and allocating rank by progressive filtering, make the principal components even more concentrated, improving both modes.

Load-bearing premise

The load-bearing premise is that a covariance matrix computed from 256 sampled activations of one task faithfully identifies which weight directions encode that task; if those samples miss the task's characteristic activation patterns, freezing the top components will not preserve the intended knowledge and adapting them will not speed learning.

Editorial extensions

If this is right

  • In knowledge-preserved mode, fine-tuning on math, code, or instruction-following data no longer has to erase QA world knowledge: the reported TriviaQA and NQ-open scores stay well above LoRA's and full fine-tuning's.
  • In instruction-previewed mode, initialization carries task structure into the adapter, so training loss drops faster; the reported speedup over QLoRA is about 3.5x at rank 128 and 4.5x at rank 32.
  • Dynamic rank allocation lets a fixed parameter budget be spent per layer according to task sensitivity, rather than using the same rank everywhere.
  • Quantizing the residual weight after CO-SVD is safer than quantizing the original weight, because the residual concentrates in the long tail of small singular values; QCorDA reports better results than QLoRA, LoftQ, and QPiSSA in most settings.
  • The same mechanism extends to vision-language models: using covariances from image-plus-text tokens preserves zero-shot VQA ability when fine-tuning a model such as LLaVA-1.5 on a new dataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The compactness score $s(C)$ is really a per-layer statement about how much output shifts when components are removed; the same score could be reused to choose layerwise quantization bit-widths or pruning ratios, not just adapter ranks.
  • Because KPM freezes the directions associated with one task's knowledge, stacking adapters initialized from different preserved tasks might offer a rehearsal-free route to continual fine-tuning, though the paper does not test this.
  • The 256-sample covariance estimate surviving as few as 32 samples suggests the method could apply in data-scarce regimes; a natural test is whether 8 or 16 samples still orient the decomposition correctly.
  • If the covariance of activations is a stable task signature, then the same decomposition could be recomputed during training to update which directions are frozen, turning static initialization into an online allocation; the paper does not explore this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CorDA++, an extension of the CorDA method for task-aware low-rank adaptation. CorDA++ computes covariance matrices of input activations from target-task samples and uses an SVD of WC to orient the decomposition, so that task-specific capability is claimed to be compacted into the principal components. Two modes are derived: knowledge-preserved mode (KPM), which freezes the principal components and adapts the tail, and instruction-previewed mode (IPM), which adapts the principal components. The paper further introduces a compactness metric and two dynamic strategies (covariance selection and rank allocation) to improve the decomposition. Experiments across LLMs, VLMs, quantized fine-tuning, and GLUE show better downstream performance and less knowledge forgetting than LoRA and other baselines, with convergence speedups in IPM.

Significance. The paper is a strong empirical contribution to parameter-efficient fine-tuning. Its strengths include: experiments across LLaMA-2-7B/13B, LLaMA-3-8B, Gemma-2-9B, LLaVA-1.5, and RoBERTa-base; comparisons with a wide range of baselines (LoRA, AdaLoRA, DoRA, PiSSA, QLoRA, LoftQ, QPiSSA, etc.); standard deviations reported over five seeds for the main tables; and integration into the Hugging Face PEFT library, which should facilitate reproducibility. The analytical bound in Sec. IV-A is a valid and useful heuristic for guiding covariance selection and rank allocation, even if it is not used to derive a formal guarantee. The paper is honest that the compactness metric is a heuristic, and the empirical ablations (Tables VII and VIII) show that the choice of context data matters, which partially validates the central premise.

major comments (3)
  1. [Abstract and Sec. VI-C] The claim of a “4.5x speedup over QLoRA” is based on the number of training iterations needed to reach a given training loss, not on wall-clock time; Table IX shows that the wall-clock training time for QCorDA and QLoRA is identical (7.3 GPU hours). Please qualify the abstract and Sec. VI-C wording to “iteration speedup” or “convergence speedup in steps”, and if the speedup is intended to imply wall-clock efficiency, report actual per-iteration time comparisons.
  2. [Sec. III-B, Sec. III-C, Tables VII and VIII] The central premise that CO-SVD compacts task-specific capability into the top principal components is supported only indirectly: Fig. 4 demonstrates that discarding bottom components barely changes language-modeling perplexity, which is a global, task-agnostic metric, and Fig. 1 shows covariance-pattern similarity across tasks. Tables VII and VIII provide indirect evidence by showing that the choice of context data affects knowledge preservation and adaptation performance. However, a direct causal probe would strengthen the paper: e.g., compute CO-SVD with QA-context data, discard the top r components (instead of the bottom r), and measure QA benchmark accuracy to verify that the top components indeed encode the target capability. Please add such an experiment or, at minimum, acknowledge that this premise remains an assumption supported only by indirect evidence.
  3. [Algorithm 2, Sec. IV-C] The stopping conditions in Algorithm 2 cause the actual adapter parameter count to exceed the budget in KPM (break when τ′ > τ) and to fall below the budget in IPM (break when τ′ < τ). The paper states that filtering continues “until the total adapter parameter amount aligns with the target parameter budget,” but the described procedure does not guarantee alignment. Please clarify how the final rank allocation is adjusted to match the budget exactly, or state that the budget is treated as an upper/lower bound.
minor comments (6)
  1. [Table VI] Unlike the other main tables, Table VI does not report standard deviations; please add them or explain why they are omitted.
  2. [Eq. (7)] The use of a logarithmic transformation of π(C) in the compactness score is not motivated; a brief justification of why the log scale is appropriate would improve clarity.
  3. [Abstract and conclusion] The phrase “outperforms CorDA by a significant margin” is not supported by all metrics; for example, in Table I(c) the MTBench scores of CorDA and CorDA++ overlap within standard deviations. Please use a more measured formulation.
  4. [Fig. 8] The speedup labels (“~1.5x”, “~3.5x”, “~4.5x”) are read from the horizontal gap between training loss curves; please specify the reference loss level at which the speedup is measured and add a note that this is an iteration-based comparison.
  5. [Sec. III-B] For layers where d_in exceeds B*L, the covariance matrix C is rank-deficient and requires regularization; the paper describes an iterative procedure based on an ℓ₂ distance threshold but does not report the threshold value. Please provide the default threshold used in experiments.
  6. [Throughout] The relationship to the authors' prior CorDA (reference [31]) is clearly stated, but the paper would benefit from a brief summary of what is new relative to the conference version (the two dynamic strategies and the quantitative experiments) in a dedicated paragraph.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: task-aware initialization and the compactness metric are validated on held-out benchmarks rather than fitted to the evaluation.

full rationale

The derivation chain runs: (i) compute C = XX^T from 256 unlabeled samples of the task; (ii) perform SVD(WC) and reconstruct via C^{-1}; (iii) initialize adapters from selected singular components; (iv) optionally select covariance matrices and allocate ranks via the bound in Eq. (6). No step fits a parameter to the evaluation metrics: the covariance uses activation statistics only, and the compactness score is an internal upper bound rather than a trained predictor of accuracy. Claims about task-specific compaction are empirical, supported by singular-value distribution plots (Fig. 3), truncation perplexity (Fig. 4), and cross-context ablations (Table VII), and the headline results are measured on held-out benchmarks (GSM8k, HumanEval, MBPP, MTBench, TriviaQA, NQ open, WebQS, GLUE, and VQA benchmarks). The KPM setting samples covariance from NQ open training inputs and then reports NQ open retention; this is the closest thing to a same-distribution evaluation, but because the covariance uses no labels or scores, the retention numbers are not forced by construction. Self-citations, including the prior CorDA [31], provide the base method and background, but the paper restates the decomposition and validates the new dynamic strategies against independent baselines; no load-bearing claim reduces to a self-citation. No uniqueness theorem, ansatz-smuggling citation, or renaming of a known result was found. Hence no circular step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method's central mechanism depends on activation-covariance-identified task directions and a heuristic compactness bound; these are unproven but empirically motivated. Hyperparameters N, sample count, and the C-inversion regularization are hand-chosen. No new physical or model entities are introduced.

free parameters (3)
  • N (number of covariance sampling rounds) = 5 (default; tested 1, 3, 10)
    Chosen by hand via Table IX; larger N gives diminishing returns but increases pre-processing time.
  • Samples per covariance round = 256
    Default in all experiments; Appendix B notes that 32 samples work slightly worse.
  • C-regularization coefficient and inversion threshold = Not reported
    Used to force invertibility of C (Sec III-B); values are not stated, making exact reproduction difficult.
assumptions (4)
  • domain assumption The covariance C = XX^T of input activations from a small sample of a target task captures the directions in weight space most relevant to that task.
    Motivates CO-SVD (Sec III-B, Fig 1); supported only by qualitative covariance heatmaps, not a formal proof.
  • ad hoc to paper The principal components of W C correspond to the task capability, so freezing them preserves knowledge and adapting them accelerates learning.
    Used in Sec V-A/V-B to justify KPM and IPM; the correspondence is inferred from downstream benchmark behavior, not derived.
  • ad hoc to paper The bound ||DeltaY||_1 <= pi(C) * sigma_{-r} is an adequate proxy for the actual loss from truncation, so minimizing it selects better covariance matrices and rank allocations.
    Sec IV-A derives an upper bound; Sec IV-B/IV-C use it as a ranking score without proving tightness or a direct link to final fine-tuning performance.
  • domain assumption C can be made invertible through dynamic diagonal regularization without materially changing the decomposition.
    Sec III-B; the regularization coefficient and convergence threshold are unspecified, and the effect on singular vectors is not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence." pith.science (2026). https://pith.science/paper/T2SUDJCK

@misc{pith2026250613187,
  author       = {Pith},
  title        = {Pith review of: Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T2SUDJCK}},
  note         = {Machine review of arXiv:2506.13187}
}
read the original abstract

Conventional low-rank adaptation methods build adapters without considering data context, leading to sub-optimal fine-tuning performance and severe forgetting of inherent world knowledge. In this paper, we propose context-oriented decomposition adaptation (CorDA), a novel method that initializes adapters in a task-aware manner. Concretely, we develop context-oriented singular value decomposition, where we collect covariance matrices of input activations for each linear layer using sampled data from the target task, and apply SVD to the product of weight matrix and its corresponding covariance matrix. By doing so, the task-specific capability is compacted into the principal components. Thanks to the task awareness, our method enables two optional adaptation modes, knowledge-preserved mode (KPM) and instruction-previewed mode (IPM), providing flexibility to choose between freezing the principal components to preserve their associated knowledge or adapting them to better learn a new task. We further develop CorDA++ by deriving a metric that reflects the compactness of task-specific principal components, and then introducing dynamic covariance selection and dynamic rank allocation strategies based on the same metric. The two strategies provide each layer with the most representative covariance matrix and a proper rank allocation. Experimental results show that CorDA++ outperforms CorDA by a significant margin. CorDA++ in KPM not only achieves better fine-tuning performance than LoRA, but also mitigates the forgetting of pre-trained knowledge in both large language models and vision language models. For IPM, our method exhibits faster convergence, \emph{e.g.,} 4.5x speedup over QLoRA, and improves adaptation performance in various scenarios, outperforming strong baseline methods. Our method has been integrated into the PEFT library developed by Hugging Face.

Figures

Figures reproduced from arXiv: 2506.13187 by the authors.

Figure 1
Figure 1. Covariance matrix visualization for 4 different input [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An illustration of our proposed context-oriented SVD, which leverages data context from the target task and performs [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Singular value distribution of plain SVD, ASVD, and our proposed context-oriented SVD (CO-SVD). The embedded [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Perplexity (lower is better) on (a) Wikitext-2 and (b) Penn TreeBank (PTB) after decomposing each weight matrix in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Perplexity at different compression ratios by progressively removing bottom components for the original CO-SVD, CO [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: In (a) knowledge-preserved adaptation mode, we use the bottom [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Zero-shot and adaptation performances of LLaVA-1.5 using full fine-tuning, LoRA, and CorDA++ (KPM) on (a) [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: The training loss curves on MetaMath of full fine-tuning, QLoRA, LoftQ, QPiSSA, and QCorDA with (a) rank 128 [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Singular value distribution of plain SVD, ASVD, and our proposed context-oriented SVD (CO-SVD). The embedded [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Covariance matrix visualization results of the “self [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

117 extracted references · 45 canonical work pages

  1. [23]

    NOLA: Compressing loRA using linear combination of random basis,

    S. A. Koohpayegani, N. K. L, P. Nooralinejad, S. Kolouri, and H. Pirsiavash, “NOLA: Compressing loRA using linear combination of random basis,” in ICLR, 2024

    2024

  2. [27]

    Pissa: Principal singular values and singular vectors adaptation of large language models,

    F. Meng, Z. Wang, and M. Zhang, “Pissa: Principal singular values and singular vectors adaptation of large language models,” in NeurIPS, 2024

    2024

  3. [113]

    Lora-ga: Low-rank adaptation with gradient approximation,

    S. Wang, L. Yu, and J. Li, “Lora-ga: Low-rank adaptation with gradient approximation,” in NeurIPS, vol. 37, 2024, pp. 54 905–54 931

    work page 2024

  4. [1]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  5. [2]

    Improving language understanding by generative pre-training,

    A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al., “Improving language understanding by generative pre-training,” 2018

    2018

  6. [3]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” in NeurIPS, vol. 33, 2020, pp. 1877– 1901

    2020

  7. [4]

    Comet: Commonsense transformers for automatic knowledge graph construction,

    A. Bosselut, H. Rashkin, M. Sap, C. Malaviya, A. Celikyilmaz, and Y . Choi, “Comet: Commonsense transformers for automatic knowledge graph construction,” arXiv preprint arXiv:1906.05317 , 2019

    arXiv 1906

  8. [5]

    Fine-tuning language models from human preferences,

    D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving, “Fine-tuning language models from human preferences,” arXiv preprint arXiv:1909.08593 , 2019

    arXiv 1909

Show all 117 references
  1. [6]

    Parameter-efficient transfer learning for nlp,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” in ICML. PMLR, 2019, pp. 2790–2799

    2019

  2. [7]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in ICLR, 2022

    2022

  3. [8]

    Towards a unified view of parameter-efficient transfer learning,

    J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig, “Towards a unified view of parameter-efficient transfer learning,” in ICLR, 2022

    2022

  4. [9]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” in EMNLP, 2021, pp. 3045–3059

    2021

  5. [10]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), ...

    2021

  6. [11]

    Residual prompt tuning: improving prompt tuning with residual reparameterization,

    A. Razdaibiedina, Y . Mao, M. Khabsa, M. Lewis, R. Hou, J. Ba, and A. Almahairi, “Residual prompt tuning: improving prompt tuning with residual reparameterization,” in ACL, 2023, pp. 6740–6757

    2023

  7. [12]

    Adaptive budget allocation for parameter-efficient fine- tuning,

    Q. Zhang, M. Chen, A. Bukharin, P. He, Y . Cheng, W. Chen, and T. Zhao, “Adaptive budget allocation for parameter-efficient fine- tuning,” in ICLR, 2023

    2023

  8. [13]

    Dylora: Parameter efficient tuning of pre-trained models using dynamic search- free low-rank adaptation,

    M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi, “Dylora: Parameter efficient tuning of pre-trained models using dynamic search- free low-rank adaptation,” arXiv preprint arXiv:2210.07558 , 2022

    2022 arXiv

  9. [14]

    Increlora: Incremental parameter allocation method for parameter-efficient fine- tuning,

    F. Zhang, L. Li, J. Chen, Z. Jiang, B. Wang, and Y . Qian, “Increlora: Incremental parameter allocation method for parameter-efficient fine- tuning,” arXiv preprint arXiv:2308.12043 , 2023

    2023 arXiv

  10. [15]

    Dora: Weight-decomposed low-rank adaptation,

    S.-Y . Liu, C.-Y . Wang, H. Yin, P. Molchanov, Y .-C. F. Wang, K.- T. Cheng, and M.-H. Chen, “Dora: Weight-decomposed low-rank adaptation,” in ICML, 2024

    2024

  11. [16]

    Lora+: efficient low rank adaptation of large models,

    S. Hayou, N. Ghosh, and B. Yu, “Lora+: efficient low rank adaptation of large models,” in ICML, 2024

    2024

  12. [17]

    Vera: Vector-based random matrix adaptation,

    D. J. Kopiczko, T. Blankevoort, and Y . M. Asano, “Vera: Vector-based random matrix adaptation,” in ICLR, 2024

    2024

  13. [18]

    Tied-lora: Enhac- ing parameter efficiency of lora with weight tying,

    A. Renduchintala, T. Konuk, and O. Kuchaiev, “Tied-lora: Enhac- ing parameter efficiency of lora with weight tying,” arXiv preprint arXiv:2311.09578, 2023

    2023 arXiv

  14. [19]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” in NeurIPS, vol. 36, 2023

    2023

  15. [20]

    QA-loRA: Quantization-aware low-rank adaptation of large language models,

    Y . Xu, L. Xie, X. Gu, X. Chen, H. Chang, H. Zhang, Z. Chen, X. ZHANG, and Q. Tian, “QA-loRA: Quantization-aware low-rank adaptation of large language models,” in ICLR, 2024

    2024

  16. [21]

    Loftq: LoRA-fine-tuning-aware quantization for large language mod- els,

    Y . Li, Y . Yu, C. Liang, N. Karampatziakis, P. He, W. Chen, and T. Zhao, “Loftq: LoRA-fine-tuning-aware quantization for large language mod- els,” in ICLR, 2024

    2024

  17. [22]

    LoRAPrune: Structured pruning meets low-rank parameter-efficient fine-tuning,

    M. Zhang, H. Chen, C. Shen, Z. Yang, L. Ou, X. Yu, and B. Zhuang, “LoRAPrune: Structured pruning meets low-rank parameter-efficient fine-tuning,” in Findings of the Association for Computational Lin- guistics, 2024, pp. 3013–3026

    2024

  18. [24]

    VB-loRA: Extreme parameter efficient fine- tuning with vector banks,

    Y . Li, S. Han, and S. Ji, “VB-loRA: Extreme parameter efficient fine- tuning with vector banks,” in NeurIPS, 2024

    2024

  19. [25]

    The impact of initialization on lora finetuning dynamics,

    S. Hayou, N. Ghosh, and B. Yu, “The impact of initialization on lora finetuning dynamics,” in NeurIPS, vol. 37, 2024, pp. 117 015–117 040

    2024

  20. [26]

    Lora-xs: Low-rank adaptation with extremely small number of parameters,

    K. Bałazy, M. Banaei, K. Aberer, and J. Tabor, “Lora-xs: Low-rank adaptation with extremely small number of parameters,” arXiv preprint arXiv:2405.17604, 2024

    2024 arXiv

  21. [28]

    Latent retrieval for weakly supervised open domain question answering,

    K. Lee, M.-W. Chang, and K. Toutanova, “Latent retrieval for weakly supervised open domain question answering,” in ACL, 2019

    2019

  22. [29]

    TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion,

    M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer, “TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion,” in ACL, 2017

    2017

  23. [30]

    Metamath: Bootstrap your own mathematical questions for large language models,

    L. Yu, W. Jiang, H. Shi, J. YU, Z. Liu, Y . Zhang, J. Kwok, Z. Li, A. Weller, and W. Liu, “Metamath: Bootstrap your own mathematical questions for large language models,” in ICLR, 2024

    2024

  24. [31]

    Corda: Context-oriented decomposition adaptation of large language models for task-aware parameter-efficient fine-tuning,

    Y . Yang, X. Li, Z. Zhou, S. Song, J. Wu, L. Nie, and B. Ghanem, “Corda: Context-oriented decomposition adaptation of large language models for task-aware parameter-efficient fine-tuning,” in NeurIPS, vol. 37, 2024, pp. 71 768–71 791

    2024

  25. [32]

    Towards theoretically inspired neural initialization optimization,

    Y . Yang, H. Wang, H. Yuan, and Z. Lin, “Towards theoretically inspired neural initialization optimization,” in NeurIPS, vol. 35, 2022, pp. 18 983–18 995

    2022

  26. [33]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano et al., “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  27. [34]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  28. [35]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021

    2021 arXiv

  29. [36]

    Judging llm-as-a-judge with mt- bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al., “Judging llm-as-a-judge with mt- bench and chatbot arena,” in NeurIPS, vol. 36, 2023

    2023

  30. [37]

    Semantic parsing on freebase from question-answer pairs,

    J. Berant, A. Chou, R. Frostig, and P. Liang, “Semantic parsing on freebase from question-answer pairs,” in EMNLP, 2013, pp. 1533– 1544

    2013

  31. [38]

    8-bit optimiz- ers via block-wise quantization,

    T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer, “8-bit optimiz- ers via block-wise quantization,” in ICLR, 2022

    2022

  32. [39]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” in NeurIPS, vol. 36, 2023, pp. 34 892–34 916

    2023

  33. [40]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” in CVPR, 2024, pp. 26 296–26 306

    2024

  34. [41]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  35. [42]

    Renaissance: A survey into ai text-to-image generation in the era of large model,

    F. Bie, Y . Yang, Z. Zhou, A. Ghanem, M. Zhang, Z. Yao, X. Wu, C. Holmes, P. Golnari, D. A. Clifton, Y . He, D. Tao, and S. L. Song, “Renaissance: A survey into ai text-to-image generation in the era of large model,” IEEE Transactions on Pattern Analysis and Machine Intelligen...

    2025

  36. [43]

    Galore: Memory-efficient llm training by gradient low-rank projec- tion,

    J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y . Tian, “Galore: Memory-efficient llm training by gradient low-rank projec- tion,” in ICML, 2024

    2024

  37. [44]

    Towards interpretable deep local learning with successive gradient reconciliation,

    Y . Yang, X. Li, M. Alfarra, H. Hammoud, A. Bibi, P. Torr, and B. Ghanem, “Towards interpretable deep local learning with successive gradient reconciliation,” in ICML, 2024

    2024

  38. [45]

    Parameter-efficient fine-tuning of large-scale pre-trained language models,

    N. Ding, Y . Qin, G. Yang, F. Wei, Z. Yang, Y . Su, S. Hu, Y . Chen, C.- M. Chan, W. Chen et al., “Parameter-efficient fine-tuning of large-scale pre-trained language models,” Nature Machine Intelligence , vol. 5, no. 3, pp. 220–235, 2023

    2023

  39. [46]

    Parameter- efficient fine-tuning methods for pretrained language models: A critical review and assessment,

    L. Xu, H. Xie, S.-Z. J. Qin, X. Tao, and F. L. Wang, “Parameter- efficient fine-tuning methods for pretrained language models: A critical review and assessment,” arXiv preprint arXiv:2312.12148 , 2023

    2023 arXiv

  40. [47]

    Conditional adapters: Parameter-efficient transfer learning with fast inference,

    T. Lei, J. Bai, S. Brahma, J. Ainslie, K. Lee, Y . Zhou, N. Du, V . Y . Zhao, Y . Wu, B. Li et al. , “Conditional adapters: Parameter-efficient transfer learning with fast inference,” in NeurIPS, 2023

    2023

  41. [48]

    Parameter- efficient multi-task fine-tuning for transformers via shared hypernet- works,

    R. K. Mahabadi, S. Ruder, M. Dehghani, and J. Henderson, “Parameter- efficient multi-task fine-tuning for transformers via shared hypernet- works,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confe...

    2021

  42. [49]

    Adapter- fusion: Non-destructive task composition for transfer learning,

    J. Pfeiffer, A. Kamath, A. R ¨uckl´e, K. Cho, and I. Gurevych, “Adapter- fusion: Non-destructive task composition for transfer learning,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , 2021, pp. 487–503. 15

    2021

  43. [50]

    Compacter: Ef- ficient low-rank hypercomplex adapter layers,

    R. Karimi Mahabadi, J. Henderson, and S. Ruder, “Compacter: Ef- ficient low-rank hypercomplex adapter layers,” in NeurIPS, vol. 34, 2021, pp. 1022–1035

    2021

  44. [51]

    SPT: Learning to selectively insert prompts for better prompt tuning,

    W. Zhu and M. Tan, “SPT: Learning to selectively insert prompts for better prompt tuning,” in EMNLP, 2023

    2023

  45. [52]

    Measuring the intrinsic dimension of objective landscapes,

    C. Li, H. Farkhoor, R. Liu, and J. Yosinski, “Measuring the intrinsic dimension of objective landscapes,” in ICLR, 2018

    2018

  46. [53]

    Intrinsic dimension- ality explains the effectiveness of language model fine-tuning,

    A. Aghajanyan, L. Zettlemoyer, and S. Gupta, “Intrinsic dimension- ality explains the effectiveness of language model fine-tuning,” arXiv preprint arXiv:2012.13255, 2020

    2012 arXiv

  47. [54]

    One-for-all: Generalized lora for parameter-efficient fine-tuning,

    A. Chavan, Z. Liu, D. Gupta, E. Xing, and Z. Shen, “One-for-all: Generalized lora for parameter-efficient fine-tuning,” arXiv preprint arXiv:2306.07967, 2023

    2023 arXiv

  48. [55]

    Controlling text-to-image diffusion by orthogonal finetuning,

    Z. Qiu, W. Liu, H. Feng, Y . Xue, Y . Feng, Z. Liu, D. Zhang, A. Weller, and B. Sch ¨olkopf, “Controlling text-to-image diffusion by orthogonal finetuning,” in NeurIPS, vol. 36, 2023, pp. 79 320–79 362

    2023

  49. [56]

    LQ-loRA: Low-rank plus quantized matrix decomposition for efficient language model finetuning,

    H. Guo, P. Greengard, E. Xing, and Y . Kim, “LQ-loRA: Low-rank plus quantized matrix decomposition for efficient language model finetuning,” in ICLR, 2024

    2024

  50. [57]

    Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications,

    Q. Liu, X. Wu, X. Zhao, Y . Zhu, D. Xu, F. Tian, and Y . Zheng, “Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications,” arXiv preprint arXiv:2310.18339 , 2023

    2023 arXiv

  51. [58]

    Loramoe: Alleviate world knowledge for- getting in largelanguage models via moe-style plugin,

    S. Dou, E. Zhou, Y . Liu, S. Gao, J. Zhao, W. Shen, Y . Zhou, Z. Xi, X. Wang, X. Fan et al. , “Loramoe: Alleviate world knowledge for- getting in largelanguage models via moe-style plugin,” arXiv preprint arXiv:2312.09979, 2023

    2023 arXiv

  52. [59]

    Milora: Harnessing minor singular components for parameter-efficient llm finetuning,

    H. Wang, Y . Li, S. Wang, G. Chen, and Y . Chen, “Milora: Harnessing minor singular components for parameter-efficient llm finetuning,” arXiv preprint arXiv:2406.09044 , 2024

    2024 arXiv

  53. [60]

    Awq: Activation-aware weight quanti- zation for llm compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “Awq: Activation-aware weight quanti- zation for llm compression and acceleration,” in MLSys, 2024

    2024

  54. [61]

    Asvd: Activation-aware singular value decomposition for compressing large language models,

    Z. Yuan, Y . Shang, Y . Song, Q. Wu, Y . Yan, and G. Sun, “Asvd: Activation-aware singular value decomposition for compressing large language models,” arXiv preprint arXiv:2312.05821 , 2023

    2023 arXiv

  55. [62]

    Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models,

    C. Lee, J. Jin, T. Kim, H. Kim, and E. Park, “Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models,” in AAAI, vol. 38, no. 12, 2024, pp. 13 355–13 364

    2024

  56. [63]

    An empirical investigation of catastrophic forgetting in gradient-based neural networks,

    I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y . Bengio, “An empirical investigation of catastrophic forgetting in gradient-based neural networks,” arXiv preprint arXiv:1312.6211 , 2013

    2013 arXiv

  57. [64]

    icarl: Incremental classifier and representation learning,

    S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” in CVPR, 2017, pp. 2001–2010

    2017

  58. [65]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al. , “Overcoming catastrophic forgetting in neural networks,” Pro- ceedings of the national academy of sciences , vol. 114, no. 13, pp. 3521...

    2017

  59. [66]

    Continual unsupervised representation learning,

    D. Rao, F. Visin, A. Rusu, R. Pascanu, Y . W. Teh, and R. Hadsell, “Continual unsupervised representation learning,” in NeurIPS, vol. 32, 2019

    2019

  60. [67]

    Gradient episodic memory for contin- ual learning,

    D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for contin- ual learning,” in NeurIPS, vol. 30, 2017

    2017

  61. [68]

    Neural collapse inspired feature-classifier alignment for few-shot class incremental learning,

    Y . Yang, H. Yuan, X. Li, Z. Lin, P. Torr, and D. Tao, “Neural collapse inspired feature-classifier alignment for few-shot class incremental learning,” in ICLR, 2023

    2023

  62. [69]

    Neural collapse terminus: A unified solution for class in- cremental learning and its variants,

    Y . Yang, H. Yuan, X. Li, J. Wu, L. Zhang, Z. Lin, P. Torr, D. Tao, and B. Ghanem, “Neural collapse terminus: A unified solution for class in- cremental learning and its variants,” arXiv preprint arXiv:2308.01746 , 2023

    2023 arXiv

  63. [70]

    Enhancing online continual learning with plug-and-play state space model and class-conditional mixture of discretization,

    S. Liu, Y . Yang, X. Li, D. A. Clifton, and B. Ghanem, “Enhancing online continual learning with plug-and-play state space model and class-conditional mixture of discretization,” in CVPR, 2025

    2025

  64. [71]

    Learning without forgetting,

    Z. Li and D. Hoiem, “Learning without forgetting,” IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 12, pp. 2935– 2947, 2017

    2017

  65. [72]

    Learning a unified classifier incrementally via rebalancing,

    S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin, “Learning a unified classifier incrementally via rebalancing,” in CVPR, 2019, pp. 831–839

    2019

  66. [73]

    Learning to learn without forgetting by maximizing transfer and minimizing interference,

    M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y . Tu, , and G. Tesauro, “Learning to learn without forgetting by maximizing transfer and minimizing interference,” in ICLR, 2019

    2019

  67. [74]

    Der: Dynamically expandable representation for class incremental learning,

    S. Yan, J. Xie, and X. He, “Der: Dynamically expandable representation for class incremental learning,” in CVPR, 2021, pp. 3014–3023

    2021

  68. [75]

    Con- tinual learning for large language models: A survey,

    T. Wu, L. Luo, Y .-F. Li, S. Pan, T.-T. Vu, and G. Haffari, “Con- tinual learning for large language models: A survey,” arXiv preprint arXiv:2402.01364, 2024

    2024 arXiv

  69. [76]

    Continual instruction tuning for large multimodal models,

    J. He, H. Guo, M. Tang, and J. Wang, “Continual instruction tuning for large multimodal models,” arXiv preprint arXiv:2311.16206, 2023

    2023 arXiv

  70. [77]

    Investi- gating the catastrophic forgetting in multimodal large language model fine-tuning,

    Y . Zhai, S. Tong, X. Li, M. Cai, Q. Qu, Y . J. Lee, and Y . Ma, “Investi- gating the catastrophic forgetting in multimodal large language model fine-tuning,” in Conference on Parsimony and Learning (Proceedings Track), 2024

    2024

  71. [78]

    Fine-tuned language models are continual learners,

    T. Scialom, T. Chakrabarty, and S. Muresan, “Fine-tuned language models are continual learners,” in EMNLP, 2022

    2022

  72. [79]

    Continual pre-training of large language models: How to re-warm your model?

    K. Gupta, B. Th ´erien, A. Ibrahim, M. L. Richter, Q. G. Anthony, E. Belilovsky, I. Rish, and T. Lesort, “Continual pre-training of large language models: How to re-warm your model?” in Workshop on Efficient Systems for Foundation Models @ ICML2023 , 2023

    2023

  73. [80]

    Simple and scalable strategies to continu- ally pre-train large language models,

    A. Ibrahim, B. Th ´erien, K. Gupta, M. L. Richter, Q. Anthony, T. Lesort, E. Belilovsky, and I. Rish, “Simple and scalable strategies to continu- ally pre-train large language models,”arXiv preprint arXiv:2403.08763, 2024

    2024 arXiv

  74. [81]

    Llama pro: Progressive llama with block expansion,

    C. Wu, Y . Gan, Y . Ge, Z. Lu, J. Wang, Y . Feng, P. Luo, and Y . Shan, “Llama pro: Progressive llama with block expansion,” arXiv preprint arXiv:2401.02415, 2024

    2024 arXiv

  75. [82]

    Composing parameter-efficient modules with arithmetic operation,

    J. Zhang, J. Liu, J. He et al., “Composing parameter-efficient modules with arithmetic operation,” NeurIPS, vol. 36, pp. 12 589–12 610, 2023

    2023

  76. [83]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch,

    L. Yu, B. Yu, H. Yu, F. Huang, and Y . Li, “Language models are super mario: Absorbing abilities from homologous models as a free lunch,” in ICML, 2024

    2024

  77. [84]

    Model tailor: Mitigating catastrophic forgetting in multi-modal large language models,

    D. Zhu, Z. Sun, Z. Li, T. Shen, K. Yan, S. Ding, K. Kuang, and C. Wu, “Model tailor: Mitigating catastrophic forgetting in multi-modal large language models,” in ICML, 2024

    2024

  78. [85]

    Language model compression with weighted low-rank factorization,

    Y .-C. Hsu, T. Hua, S. Chang, Q. Lou, Y . Shen, and H. Jin, “Language model compression with weighted low-rank factorization,” in ICLR, 2022

    2022

  79. [86]

    SVD-LLM: Truncation- aware singular value decomposition for large language model compres- sion,

    X. Wang, Y . Zheng, Z. Wan, and M. Zhang, “SVD-LLM: Truncation- aware singular value decomposition for large language model compres- sion,” in ICLR, 2025

    2025

  80. [87]

    Optimizing singular spectrum for large language model compression,

    D. Li, T. Shen, Y . Zhou, B. Yang, Z. Liu, M. Yang, B. Ghanem, Y . Yang, Y . Zhong, and M.-H. Yang, “Optimizing singular spectrum for large language model compression,” arXiv preprint arXiv:2502.15092, 2025

    2025 arXiv

  81. [88]

    Exploring post-training quantization in llms from comprehensive study to low rank compensa- tion,

    Z. Yao, X. Wu, C. Li, S. Youn, and Y . He, “Exploring post-training quantization in llms from comprehensive study to low rank compensa- tion,” in AAAI, vol. 38, no. 17, 2024, pp. 19 377–19 385

    2024

  82. [89]

    SVDQuant: Absorbing outliers by low-rank component for 4-bit diffusion models,

    M. Li, Y . Lin, Z. Zhang, T. Cai, J. Guo, X. Li, E. Xie, C. Meng, J.-Y . Zhu, and S. Han, “SVDQuant: Absorbing outliers by low-rank component for 4-bit diffusion models,” in ICLR, 2025

    2025

  83. [90]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,

    K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in ICCV, 2015, pp. 1026–1034

    2015

  84. [91]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843 , 2016

    2016 arXiv

  85. [92]

    Building a large annotated corpus of english: The penn treebank,

    M. Marcus, B. Santorini, and M. A. Marcinkiewicz, “Building a large annotated corpus of english: The penn treebank,” Computational linguistics, vol. 19, no. 2, pp. 313–330, 1993

    1993

  86. [93]

    Natural questions: a benchmark for question answering research,

    T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee et al., “Natural questions: a benchmark for question answering research,” Transactions of the Association for Computational Linguistics , vol. 7, pp. 453–466, 2019

    2019

  87. [94]

    Gpt3.int8(): 8-bit matrix multiplication for transformers at scale,

    T. Dettmers, M. Lewis, Y . Belkada, and L. Zettlemoyer, “Gpt3.int8(): 8-bit matrix multiplication for transformers at scale,” in NeurIPS, 2022, pp. 30 318–30 332

    2022

  88. [95]

    Smoothquant: Accurate and efficient post-training quantization for large language models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” in ICML. PMLR, 2023, pp. 38 087–38 099

    2023

  89. [96]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  90. [97]

    Gemma 2: Improving open language models at a practical size,

    G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ram ´e et al. , “Gemma 2: Improving open language models at a practical size,” arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  91. [98]

    Opencodeinterpreter: Integrating code generation with execution and refinement,

    T. Zheng, G. Zhang, T. Shen, X. Liu, B. Y . Lin, J. Fu, W. Chen, and X. Yue, “Opencodeinterpreter: Integrating code generation with execution and refinement,” arXiv preprint arXiv:2402.14658 , 2024. 16

    2024 arXiv

  92. [99]

    WizardLM: Empowering large pre-trained language models to follow complex instructions,

    C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, Q. Lin, and D. Jiang, “WizardLM: Empowering large pre-trained language models to follow complex instructions,” in ICLR, 2024

    2024

  93. [100]

    Judging LLM-as-a-judge with MT-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Sto- ica, “Judging LLM-as-a-judge with MT-bench and chatbot arena,” in Thirty-seventh Conference on Neural Information Processing Systems Datasets a...

    2023

  94. [101]

    Ok-vqa: A visual question answering benchmark requiring external knowledge,

    K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi, “Ok-vqa: A visual question answering benchmark requiring external knowledge,” in CVPR, 2019, pp. 3195–3204

    2019

  95. [102]

    Pmc-vqa: Visual instruction tuning for medical visual question an- swering,

    X. Zhang, C. Wu, Z. Zhao, W. Lin, Y . Zhang, Y . Wang, and W. Xie, “Pmc-vqa: Visual instruction tuning for medical visual question an- swering,” arXiv preprint arXiv:2305.10415 , 2023

    2023 arXiv

  96. [103]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering,

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” in CVPR, 2017, pp. 6904–6913

    2017

  97. [104]

    Vizwiz grand challenge: Answering visual questions from blind people,

    D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham, “Vizwiz grand challenge: Answering visual questions from blind people,” in CVPR, 2018, pp. 3608–3617

    2018

  98. [105]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering,

    D. A. Hudson and C. D. Manning, “Gqa: A new dataset for real-world visual reasoning and compositional question answering,” in CVPR, 2019, pp. 6700–6709

    2019

  99. [106]

    Learn to explain: Multimodal reasoning via thought chains for science question answering,

    P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan, “Learn to explain: Multimodal reasoning via thought chains for science question answering,” in NeurIPS, vol. 35, 2022, pp. 2507–2521

    2022

  100. [107]

    Towards vqa models that can read,

    A. Singh, V . Natarajan, M. Shah, Y . Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach, “Towards vqa models that can read,” in CVPR, 2019, pp. 8317–8326

    2019

  101. [108]

    Mmbench: Is your multi-modal model an all- around player?

    Y . Liu, H. Duan, Y . Zhang, B. Li, S. Zhang, W. Zhao, Y . Yuan, J. Wang, C. He, Z. Liu et al. , “Mmbench: Is your multi-modal model an all- around player?” in ECCV. Springer, 2024, pp. 216–233

    2024

  102. [109]

    Internlm-xcomposer: A vision- language large model for advanced text-image comprehension and composition,

    P. Zhang, X. Dong, B. Wang, Y . Cao, C. Xu, L. Ouyang, Z. Zhao, H. Duan, S. Zhang, S. Ding et al. , “Internlm-xcomposer: A vision- language large model for advanced text-image comprehension and composition,” arXiv preprint arXiv:2309.15112 , 2023

    2023 arXiv

  103. [110]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019

    1907 arXiv

  104. [111]

    GLUE: A multi-task benchmark and analysis platform for natural language understanding,

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in ICLR, 2019

    2019

  105. [112]

    Lora-fa: Memory- efficient low-rank adaptation for large language models fine-tuning,

    L. Zhang, L. Zhang, S. Shi, X. Chu, and B. Li, “Lora-fa: Memory- efficient low-rank adaptation for large language models fine-tuning,” arXiv preprint arXiv:2308.03303 , 2023

    2023 arXiv

  106. [114]

    KaSA: Knowledge- aware singular-value adaptation of large language models,

    F. Wang, J. Jiang, C. Park, S. Kim, and J. Tang, “KaSA: Knowledge- aware singular-value adaptation of large language models,” in ICLR, 2025

    2025

  107. [115]

    Parameter-efficient fine-tuning with discrete fourier transform,

    Z. Gao, Q. Wang, A. Chen, Z. Liu, B. Wu, L. Chen, and J. Li, “Parameter-efficient fine-tuning with discrete fourier transform,” in ICML, 2024

    2024

  108. [116]

    BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language- models,

    E. Ben Zaken, Y . Goldberg, and S. Ravfogel, “BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language- models,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , 2022

    2022

  109. [117]

    AdapterDrop: On the efficiency of adapters in transformers,

    A. R ¨uckl´e, G. Geigle, M. Glockner, T. Beck, J. Pfeiffer, N. Reimers, and I. Gurevych, “AdapterDrop: On the efficiency of adapters in transformers,” in EMNLP, 2021, pp. 7930–7946. 17 APPENDIX A EXPERIMENT DETAILS Baseline method introduction. We introduce the baseline method...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.