Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

CoLA: Collaborative Low-Rank Adaptation

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By freeing the low-rank update from the rigid one-to-one pairing of A and B matrices and initializing every factor from the pretrained weights' principal singular vectors, CoLA claims consistent gains over existing PEFT methods that grow…

desk verdict A useful empirical PEFT paper whose central framing overstates novelty—CoLA⊺ and CoLA† are rank-r LoRA reparameterizations, and the headline gains are not yet cleanly separated from parameter count—but the #A<#B finding and open code make it worth referee time. read the letter →

arxiv 2505.15471 v1 pith:FZO2GYDJ submitted 2025-05-21 cs.CL

Yiyun Zhou , Chang Yao , Jingyuan Chen This is my paper
classification cs.CL
keywords parameter-efficientfine-tuninglow-rankadaptationLoRAsingularvaluedecompositionPiSSAinitializationscarcedatamulti-tasklearningasymmetricarchitecture
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fine-tuning large language models with only a few hundred labeled examples is fragile: standard LoRA and its variants bind the two low-rank factors A and B into a fixed shape, so they either cannot separate shared knowledge from task-specific detail or collapse under noise. The paper's central claim is that freeing the architecture to use any number of A matrices and any number of B matrices, together with an SVD-based initialization, recovers robust gains exactly where data is scarce. CoLA, the proposed adapter, outperforms LoRA, PiSSA, DoRA, HydraLoRA, and MoE-style baselines on single-domain and multi-task benchmarks for Llama-3.1-8B and Llama-3.2-3B, with margins that widen as samples drop from 1000 toward 200. Along the way the paper extracts a design rule: for a fixed parameter budget, more B matrices than A matrices is better, because the A side learns shared commonality and the B side learns per-component diversity.

What carries the argument

The machinery is a many-to-many low-rank adapter: instead of one pair (A,B), CoLA maintains M matrices on the input side and N on the output side, with the update written as a combination of their products (sum-of-B times sum-of-A for full collaboration, random pairing for stochastic collaboration, and a hybrid mix). A second piece is the extended PiSSA initialization: the singular value decomposition of each pretrained weight is split into principal and residual parts, and the principal singular vectors are divided evenly among all A and B matrices so that the adapter starts aligned with the full-rank update direction. The three collaboration strategies are what test the quantitative relationship between A and B, and the paper uses them to establish the #A < #B design principle.

What would settle it

Compare CoLA (#A=1,#B=3 and #A=2,#B=3, rank 8) against PiSSA at rank 16 and rank 24 on the same generality benchmark, which equalizes trainable parameters; if matched-parameter PiSSA matches or exceeds CoLA's 58.04 and 58.21 scores, the case for the architecture's specific contribution collapses.

Watch

Extended reading notes

Core claim

The paper discovers that the numerical relationship between the number of A and B matrices in low-rank adapters is itself a design axis. Existing architectures fix this relationship (LoRA: 1-1; LoRA+MoE: N-N; HydraLoRA/MTL-LoRA: 1-N), and the paper argues this rigidity is what makes them brittle when samples are scarce. CoLA sets #A=M and #B=N, initializes each matrix by evenly splitting the principal singular vectors and values of the pretrained weight (extending PiSSA), and couples the matrices through one of three collaboration rules. Empirically, CoLA and its fully-collaborative variant CoLA⊺ consistently beat all baselines, and the authors show that the benefit of increasing B outweighs increasing A, so #A < #B is the recommended configuration. The paper also reports that the extended PiSSA initialization is essential: without it, CoLA does not beat LoRA in low-sample settings.

Load-bearing premise

The paper assumes the performance gains come from the flexible multi-matrix architecture and its collaboration rules, rather than from the larger number of trainable parameters or from the SVD-based initialization alone.

Editorial extensions

If this is right

  • On zero-shot evaluations with Llama-3.1-8B, CoLA and CoLA⊺ reach 58.04 and 58.21 on the generality benchmark, versus 54.72 for PiSSA and 50.36 for LoRA r=8, with the gap growing in low-sample regimes.
  • The #A < #B design rule gives a practical recipe: when allocating a fixed parameter budget, prefer more output-side B matrices over more input-side A matrices.
  • The extended PiSSA initialization is load-bearing: removing it drops CoLA below LoRA in the 200-300 sample range.
  • The three collaboration strategies provide an explicit energy/performance trade-off, so practitioners can pick full, random, or hybrid coupling based on compute budgets.
  • CoLA remains stable at 200 samples where standard LoRA's performance degrades sharply below 300.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's attribution would be to run PiSSA at rank 16 or 24 (matching CoLA's trainable parameters) and HydraLoRA with PiSSA initialization; if either closes the gap, the flexible architecture contributes less than the initialization and parameter count.
  • The #A < #B heuristic may transfer to other adapter families: any design where one side captures shared structure and the other captures per-expert diversity should allocate more capacity to the diversity side.
  • The bipartite-graph view of A-B connections, mentioned by the authors, suggests a continuum of unexplored coupling patterns (learned sparse graphs, capacity-constrained matchings) between the fully-connected and random extremes.
  • The failure of CoLA† (random B) and success of CoLAb† (random A) hints that stochastic routing should sample from the diversity side, a guideline that could apply to MoE routing and dropout placement in adapters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces CoLA, a LoRA variant that decouples the number of A matrices (M) from the number of B matrices (N), and proposes three collaborative strategies: fully collaborative CoLA⊺, random CoLA†, and heuristic CoLA‡. It extends PiSSA's SVD-based initialization to CoLA. Experiments on Llama-3.1-8B and Llama-3.2-3B across single-domain (generality, law, medicine, math, finance) and multi-domain benchmarks report consistent improvements over existing PEFT baselines, with claimed robustness in low-sample regimes.

Significance. The empirical study is broad: two model families, five single-domain and one multi-domain benchmark, five random seeds, and a public code/data release. The authors also include a limitations section. However, the two best-performing variants (CoLA and CoLA⊺) are mathematically equivalent to standard rank-r LoRA in representable functions; their gains are therefore an optimization/parameterization effect, not an architectural expressivity gain. The central attribution of the improvements to the 'flexible architecture' is not yet established because of confounds with parameter count and initialization. If the missing controls are supplied, the finding that a collaborative reparameterization of LoRA improves fine-tuning in low-sample settings would still be of interest.

major comments (3)
  1. [§3.3 (collaborative strategies)] The displayed update for CoLA⊺ and CoLA† is ΔW = (B1+...+BN)(A1+...+AM). Because every A_i is r×m and every B_j is n×r, this equals B A with B = (B1+...+BN) and A = (A1+...+AM), which is exactly the vanilla rank-r LoRA update of Eq. (1). Thus CoLA⊺ and CoLA† do not enlarge the class of representable functions relative to rank-r LoRA; they only reparameterize it with M+N matrices instead of two. The paper's repeated claim of a 'more flexible architecture' (Abstract, §1, §3.1) is therefore inaccurate for these two variants. The authors should either reframe the contribution as an over-parameterized factorization that affects optimization (and provide evidence for that), or restrict the expressivity claim to CoLA‡, which is a genuine sum-of-low-rank experts.
  2. [§4.2, Table 2 and Figure 3] The headline comparisons against PiSSA and LoRA r=8 are not parameter-matched: in Table 2, CoLA (#A=3,#B=2) uses 0.5325% parameters and CoLA⊺ 0.6551%, versus 0.2605% for PiSSA/LoRA r=8. The reported gains of 3–4 points could thus reflect 2–2.5× more trainable parameters. Table 8 provides a partial control by comparing CoLA (#A=1,#B=3) and CoLA⊺ (#A=2,#B=3) against PiSSA r=16/r=24, but only in the Generality domain and at sample sizes ≥1000. The paper's low-sample robustness claim is centered on Figure 3 at sample size 200, where the baselines are LoRA r=16/r=24, not PiSSA. To support the attribution to the flexible architecture, the authors should add PiSSA baselines with matched parameter counts (e.g., r=16 and r=24) in the 200-sample regime across all domains, and include PiSSA-initialized HydraLoRA/MOELoRA to rule out initialization artifacts.
  3. [§4.2, Observation 3] The rule '#A should be fewer than #B' is derived from Figure 4 by varying #A and #B at fixed rank r=8. The symmetric comparison (#A=x,#B=y vs #A=y,#B=x) is parameter-matched only if the per-matrix sizes of A and B are equal (n=m); for feed-forward layers in Llama, where the intermediate dimension differs from the hidden dimension, this does not hold. The paper does not state which modules are used or whether the conclusion survives when total parameters are held constant. Since Observation 3 is used both as a design principle and to explain CoLA†'s poor performance (Table 4 and Observation 4), it needs a controlled parameter-matched analysis.
minor comments (6)
  1. [§3.2] The SVD dimensions are inconsistent: for W ∈ R^{n×m}, Eq. (2) states U ∈ R^{m×m} and V ∈ R^{n×n}, which is transposed relative to standard convention. The shapes of A_i and B_j in Eqs. (4) and (6) should be explicitly defined to avoid ambiguity.
  2. [§3.2] The sentence 'This assumption is supported by the Eckart-Young-Mirsky theorem, which is ignored by PiSSA' is confusing: the theorem is the standard justification for PiSSA, so it is not 'ignored'. Please rephrase.
  3. [Figure 3] In the submitted PDF the figure text is garbled (font/symbol substitution), making the curves unreadable; the final version must use a properly rendered figure.
  4. [Table 2] The %Param row and the #A/#B rows are hard to parse in the extracted text; ensure the table is formatted with clear column separators in the camera-ready version.
  5. [§3.3 (CoLA†)] The description of CoLA† ('combining each matrix A with a randomly chosen matrix B') is not precise about whether the random pairing is resampled per step or fixed, and how this yields the stated sum formula. Please clarify.
  6. [§4.2, Observation 4] The energy comparison with HydraLoRA mixes different sample sizes and evaluation protocols; the claim of '<1/10th of the energy' should be explicitly labeled as not a controlled comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: CoLA's empirical claims are supported by external baselines, standard SVD theory, and imported PiSSA initialization rather than by self-referential derivation.

full rationale

The paper makes no first-principles prediction that reduces to its own inputs. The architecture (#A=M, #B=N), the extended PiSSA initialization (Eq. 6), and the three collaborative strategies are described constructively and then evaluated empirically against multiple external baselines (LoRA, PiSSA, HydraLoRA, MOELoRA, etc.). The only theorem invoked, Theorem 3.1, is the standard Eckart-Young-Mirsky optimal low-rank approximation result, which is an external mathematical fact and not a claim derived from the paper's own method. PiSSA initialization is imported from prior work (Meng et al., 2024) and is itself an external method; extending it to CoLA does not create a circular link because the extension is not used to define the evaluation outcome. Observation 3 (#A < #B is beneficial) is presented as an empirical finding from Figure 4 and Table 4, not as an assumption used to construct the results; the same holds for Observations 1, 2, and 4. The paper does contain self-citations in the related-work discussion (Zhou et al., 2024, 2025a,b; Li et al., 2025) and in Appendix A for data scarcity, but those citations are contextual and non-load-bearing: the central empirical comparisons stand on independently reported baseline results and publicly available datasets. The reviewer-identified concern that CoLA's gains may be confounded by higher parameter counts and by PiSSA initialization is a legitimate experimental-control question, but it is not circular reasoning under the definitions used here: the paper does not fit a parameter and then rename that fit as a prediction. Accordingly, the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or abstract entities are introduced. The method relies on three design hyperparameters (rank, #A, #B) and a collaborative strategy choice, all chosen by the authors and tuned per experiment. The empirical results should be interpreted conditionally on these choices.

free parameters (4)
  • LoRA rank r = 8 (with some baselines at 16, 24, 32, 64)
    Set to 8 by default for CoLA and most baselines; the central results depend on this choice.
  • M, the number of A matrices = 1 for CoLA, 2 for CoLA⊺, 3 for CoLA† and CoLA‡; explored from 1 to 5
    The number of A matrices is a hyperparameter of CoLA; Observation 3 recommends M<N.
  • N, the number of B matrices = 3 for CoLA and CoLA⊺, 2 or more for the other variants; explored from 1 to 5
    The number of B matrices is a hyperparameter; the method's performance varies with it, as shown in Figure 4.
  • Collaborative strategy (⊺, †, or ‡) = One of three strategies, with plain CoLA also reported
    The choice of strategy changes the update rule and the energy consumption; the paper claims different strategies suit different application scenarios.
assumptions (4)
  • domain assumption The low-rank update hypothesis: the fine-tuning weight change ΔW lies in a low-rank subspace of rank r (Eq. 1).
    The entire CoLA architecture builds on the LoRA assumption that task-specific updates are low-rank; if the intrinsic rank is higher, CoLA's rank-r basis cannot represent the needed update.
  • domain assumption PiSSA's assumption that initializing adapters with the top-r singular vectors and values of the frozen weight W speeds convergence and improves final performance.
    CoLA extends this initialization and the paper calls it an essential component in Observation 2, Sec. 4.2.
  • standard math The Eckart-Young-Mirsky theorem (Theorem 3.1): the optimal rank-r approximation of W is the truncated SVD.
    Used to justify using the top-r singular components for initialization.
  • domain assumption Converting generative tasks (GSM8K and BBH) into multiple-choice classification with LLM-generated distractors preserves relative method performance.
    The paper's evaluation protocol in Sec. 4.1.3 and Appendix D depends on this; the authors concede in Limitations that the conversion may conflict with the original instruction setup.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoLA: Collaborative Low-Rank Adaptation." pith.science (2026). https://pith.science/paper/FZO2GYDJ

@misc{pith2026250515471,
  author       = {Pith},
  title        = {Pith review of: CoLA: Collaborative Low-Rank Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FZO2GYDJ}},
  note         = {Machine review of arXiv:2505.15471}
}
abstract

The scaling law of Large Language Models (LLMs) reveals a power-law relationship, showing diminishing return on performance as model scale increases. While training LLMs from scratch is resource-intensive, fine-tuning a pre-trained model for specific tasks has become a practical alternative. Full fine-tuning (FFT) achieves strong performance; however, it is computationally expensive and inefficient. Parameter-efficient fine-tuning (PEFT) methods, like LoRA, have been proposed to address these challenges by freezing the pre-trained model and adding lightweight task-specific modules. LoRA, in particular, has proven effective, but its application to multi-task scenarios is limited by interference between tasks. Recent approaches, such as Mixture-of-Experts (MOE) and asymmetric LoRA, have aimed to mitigate these issues but still struggle with sample scarcity and noise interference due to their fixed structure. In response, we propose CoLA, a more flexible LoRA architecture with an efficient initialization scheme, and introduces three collaborative strategies to enhance performance by better utilizing the quantitative relationships between matrices $A$ and $B$. Our experiments demonstrate the effectiveness and robustness of CoLA, outperforming existing PEFT methods, especially in low-sample scenarios. Our data and code are fully publicly available at https://github.com/zyy-2001/CoLA.

Figures

Figures reproduced from arXiv: 2505.15471 by the authors.

Figure 1
Figure 1. The comparison between Full Fine-tuning and different LoRA variant structures. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of CoLA with three collaborative strategies. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The impact of PiSSA initialization on LoRA and CoLA based on Llama-3.1-8B in the generality domain [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The performance of Llama-3.1-8B when the number of matrices [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Energy consumption of three collaborative strategies based on Llama-3.1-8B. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Evaluation Template Hyperparameter Setting Batch Size 8 Train Epochs 5.0 Validation Size 0.1 Learning Rate 5e-5 Cutoff Length 1024 Gradient Accumulation Steps 8 Random Seed 42,43,44,45,46 Scheduler Type cosine Precision fp16 Evaluation Strategy steps Optimizer Adamw GP…

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference

    cs.LG 2025-06 conditional novelty 6.0 of 10

    MadaKV adaptively splits the KV cache budget by attention-head modality preference and compensates across layers, cutting cache memory by 80-95% and speeding decoding by 1.3-1.5x with small accuracy loss.

  2. Physical AI Governance: From Theory to Practice Across Life Cycle

    cs.AI 2026-07 conditional novelty 4.0 of 10

    A survey that organizes Physical AI governance into five principles and a five-stage lifecycle, with stage-specific operational practices.

  3. Cuff-KT: Tackling Learners' Real-time Learning Pattern Adjustment via Tuning-Free Knowledge State Guided Model Updating

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Cuff-KT generates personalized output-layer parameters for knowledge tracing models without fine-tuning, reporting AUC improvements of about 10% and 4% under intra- and inter-learner shifts.

Reference graph

Works this paper leans on

65 extracted references · 20 canonical work pages · cited by 3 Pith papers

  1. [1]

    Ahmed Agiza, Marina Neseem, and Sherief Reda. 2024. Mtlora: Low-rank adaptation approach for efficient multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16196--16205

    work page 2024

  2. [2]

    Alignment-Lab-AI. 2024. Lawyer-instruct

    work page 2024

  3. [3]

    D Araci. 2019. Finbert: Financial sentiment analysis with pre-trained language models. arXiv preprint arXiv:1908.10063

    arXiv 2019

  4. [4]

    BIG bench authors. 2023. https://openreview.net/forum?id=uyTL5Bvosj Beyond the imitation game: Quantifying and extrapolating the capabilities of language models . Transactions on Machine Learning Research

    2023

  5. [5]

    Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. Legal-bert: The muppets straight out of law school. arXiv preprint arXiv:2010.02559

    arXiv 2020

  6. [6]

    Ilias Chalkidis, Nicolas Garneau, Catalina Goanta, Daniel Katz, and Anders S gaard. 2023. https://aclanthology.org/2023.acl-long.865 L e XF iles and L egal LAMA : Facilitating E nglish multinational legal language model development . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1...

    2023

  7. [7]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  8. [8]

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. Company Blog of Databricks

    2023

Show all 65 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  2. [10]

    Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. 2024. Mixture-of-loras: An efficient multitask tuning for large language models. arXiv preprint arXiv:2403.03432

    2024 arXiv

  3. [11]

    Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. 2024. Higher layers need more lora experts. arXiv preprint arXiv:2402.08562

    2024 arXiv

  4. [12]

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. 2024. The impact of initialization on lora finetuning dynamics. arXiv preprint arXiv:2406.08447

    2024 arXiv

  5. [13]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  6. [14]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  7. [15]

    Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021. Scaling laws for transfer. arXiv preprint arXiv:2102.01293

    2021 arXiv

  8. [16]

    Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. 2006. A fast learning algorithm for deep belief nets. Neural computation, 18(7):1527--1554

    2006

  9. [17]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  10. [18]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  11. [19]

    Wenjun Ke, Jiahao Wang, Peng Wang, Jiajun Liu, Dong Nie, Guozheng Li, and Yining Li. 2025. Unveiling lora intrinsic ranks via salience analysis. Advances in Neural Information Processing Systems, 37:131575--131595

    2025

  12. [20]

    Yann LeCun, L \'e on Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278--2324

    1998

  13. [21]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021 a . The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691

    2021 arXiv

  14. [22]

    Brian Lester, Rami Al - Rfou, and Noah Constant. 2021 b . https://doi.org/10.18653/V1/2021.EMNLP-MAIN.243 The power of scale for parameter-efficient prompt tuning . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual E...

    2021 doi

  15. [23]

    Kunxi Li, Tianyu Zhan, Kairui Fu, Shengyu Zhang, Kun Kuang, Jiwei Li, Zhou Zhao, Fan Wu, and Fei Wu. 2025. Mergenet: Knowledge migration across heterogeneous models, tasks, and modalities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 4824--4832

    2025

  16. [24]

    Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. 2023. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus, 15(6)

    2023

  17. [25]

    W Lian, B Goodson, E Pentland, et al. 2023. Openorca: An open dataset of gpt augmented flan reasoning traces

    2023

  18. [26]

    Vijay Lingam, Atula Tejaswi, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Alex Dimakis, Eunsol Choi, Aleksandar Bojchevski, and Sujay Sanghavi. 2024. Svft: Parameter-efficient fine-tuning with singular vectors. arXiv preprint arXiv:2405.19597

    2024 arXiv

  19. [27]

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/0cde695b83bd186c1fd456302888454c-Abstract-Conference.html Few-shot parameter-efficient fine-tuning is better and cheaper ...

    2022

  20. [28]

    Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. 2024 a . When moe meets llms: Parameter efficient fine-tuning for multi-task medical applications. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development...

    2024

  21. [29]

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024 b . Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353

    2024 arXiv

  22. [30]

    Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. https://arxiv.org/abs/2103.10385 GPT understands, too . CoRR, abs/2103.10385

    2021 arXiv

  23. [31]

    Zhiming Liu and Chengjun Liu. 2010. Fusion of color, local spatial and global frequency information for face recognition. Pattern Recognition, 43(8):2882--2890

    2010

  24. [32]

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pages 22631--22...

    2023

  25. [33]

    Fanxu Meng, Zhaohui Wang, and Muhan Zhang. 2024. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02948

    2024 arXiv

  26. [34]

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707

    2023 arXiv

  27. [35]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  28. [36]

    Dhavalkumar Patel, Prem Timsina, Ganesh Raut, Robert Freeman, Matthew A levin, Girish N Nadkarni, Benjamin S Glicksberg, and Eyal Klang. 2024. Exploring temperature effects on large language models across various clinical tasks. medRxiv, pages 2024--07

    2024

  29. [37]

    David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2021. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350

    2021 arXiv

  30. [38]

    Branislav Pecher, Ivan Srba, and Maria Bielikova. 2024. Fine-tuning, prompting, in-context learning and instruction-tuning: How many labelled samples do we need? arXiv preprint arXiv:2402.12819

    2024

  31. [39]

    Shuai Peng, Ke Yuan, Liangcai Gao, and Zhi Tang. 2021. Mathbert: A pre-trained model for mathematical formula understanding. arXiv preprint arXiv:2105.00377

    2021 arXiv

  32. [40]

    Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, et al. 2021. Exploring universal intrinsic task subspace via prompt tuning. arXiv preprint arXiv:2110.07867

    2021 arXiv

  33. [41]

    Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. 2021. Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ digital medicine, 4(1):86

    2021

  34. [42]

    Matthew Renze and Erhan Guven. 2024. The effect of sampling temperature on problem solving in large language models. arXiv preprint arXiv:2402.05201

    2024 arXiv

  35. [43]

    Maximilian Riesenhuber and Tomaso Poggio. 1999. Hierarchical models of object recognition in cortex. Nature neuroscience, 2(11):1019--1025

    1999

  36. [44]

    a fer, Till Nicke, Henning H \

    Raphael Sch \"a fer, Till Nicke, Henning H \"o fener, Annkristin Lange, Dorit Merhof, Friedrich Feuerhake, Volkmar Schulz, Johannes Lotz, and Fabian Kiessling. 2024. Overcoming data scarcity in biomedical imaging with a foundational multi-task model. Nature Computational Scien...

    2024

  37. [45]

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929--1958

    2014

  38. [46]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261

    2022 arXiv

  39. [47]

    Xiaoyang Tan and Bill Triggs. 2010. Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE transactions on image processing, 19(6):1635--1650

    2010

  40. [48]

    Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. 2024. Hydralora: An asymmetric lora architecture for efficient fine-tuning. arXiv preprint arXiv:2404.19245

    2024 arXiv

  41. [49]

    Hoang Van. 2023. Mitigating data scarcity for large language models. arXiv preprint arXiv:2302.01806

    2023 arXiv

  42. [50]

    Neng Wang, Hongyang Yang, and Christina Dan Wang. 2023. Fingpt: Instruction tuning benchmark for open-source large language models in financial datasets. NeurIPS Workshop on Instruction Tuning and Instruction Following

    2023

  43. [51]

    Shaowen Wang, Linxi Yu, and Jian Li. 2024 a . Lora-ga: Low-rank adaptation with gradient approximation. arXiv preprint arXiv:2407.05000

    2024 arXiv

  44. [52]

    Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. 2024 b . Lora-pro: Are low-rank adapters properly optimized? arXiv preprint arXiv:2407.18242

    2024 arXiv

  45. [53]

    Yaming Yang, Dilxat Muhtar, Yelong Shen, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Denvy Deng, Feng Sun, Qi Zhang, et al. 2024 a . Mtl-lora: Low-rank adaptation for multi-task learning. arXiv preprint arXiv:2410.09437

    2024 arXiv

  46. [54]

    Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. 2024 b . Corda: Context-oriented decomposition adaptation of large language models. arXiv preprint arXiv:2406.05223

    2024 arXiv

  47. [55]

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. 2022. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12104--12113

    2022

  48. [56]

    Fangzhao Zhang and Mert Pilanci. 2024. Spectral adapter: Fine-tuning in spectral space. arXiv preprint arXiv:2405.13952

    2024 arXiv

  49. [57]

    Wenyi Zhao, Rama Chellappa, P Jonathon Phillips, and Azriel Rosenfeld. 2003. Face recognition: A literature survey. ACM computing surveys (CSUR), 35(4):399--458

    2003

  50. [58]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. http://arxiv.org/abs/2403.13372 Llamafactory: Unified efficient fine-tuning of 100+ language models . In Proceedings of the 62nd Annual Meeting of the Association for Compu...

    2024 arXiv

  51. [59]

    Yiyun Zhou, Wenkang Han, and Jingyuan Chen. 2025 a . Revisiting applicable and comprehensive knowledge tracing in large-scale data. arXiv preprint arXiv:2501.14256

    2025 arXiv

  52. [60]

    Yiyun Zhou, Zheqi Lv, Shengyu Zhang, and Jingyuan Chen. 2024. https://openreview.net/forum?id=UVaPEthRKx Cuff- KT : Tackling learners' real-time learning pattern adjustment via tuning-free knowledge state-guided model updating

    2024

  53. [61]

    Yiyun Zhou, Zheqi Lv, Shengyu Zhang, and Jingyuan Chen. 2025 b . Disentangled knowledge tracing for alleviating cognitive bias. In Proceedings of the ACM on Web Conference 2025, pages 2633--2645

    2025

  54. [62]

    Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz S \'a ez de Oc \'a riz Borde, Rickard Br \"u el Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. 2024 a . Asymmetry in low-rank adapters of foundation models. arXiv preprint arXiv:2402.16842

    2024 arXiv

  55. [63]

    Yuqi Zhu, Jia Li, Ge Li, YunFei Zhao, Zhi Jin, and Hong Mei. 2024 b . Hot or cold? adaptive temperature sampling for code generation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 437--445

    2024

  56. [64]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  57. [65]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.