Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuning

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By orthogonalizing expert output vectors with Gram-Schmidt at every token, OMoE makes a two-expert mixture of LoRA adapters match or beat eight-expert baselines while reducing tunable parameters by about 75 percent.

desk verdict A plausible PEFT idea with a confounded evaluation and a formal gap in the Gram-Schmidt step; worth refereeing but not ready as is. read the letter →

arxiv 2501.10062 v2 pith:TR5NRPWK submitted 2025-01-17 cs.LG cs.CL

Jinyuan Feng , Zhiqiang Pu , Tianyi Hu , Dongmin Li , Xiaolin Ai , Huimu Wang This is my paper · ORCID
classification cs.LGcs.CL
keywords orthogonalmixture-of-expertslow-rankadaptationparameter-efficientfine-tuningexpertdiversityGram-SchmidtorthogonalizationStiefelmanifoldlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that mixture-of-experts (MoE) versions of low-rank adaptation fail to scale because the experts learn redundant, nearly identical representations, so adding experts wastes parameters and compute. It proposes OMoE, which runs the experts' output vectors through a Gram-Schmidt orthogonalization at every forward pass, forcing the representations onto the Stiefel manifold and guaranteeing diversity by construction. The central empirical claim is that with only two orthogonal experts, OMoE matches or beats state-of-the-art PEFT baselines — LoRA, DoRA, and MixLoRA — across commonsense reasoning benchmarks while reducing tunable parameters by roughly 75 percent. The authors further claim that because the constraint is architectural rather than a regularizer, the learning objective stays unchanged, so the method preserves optimality.

What carries the argument

The load-bearing object is the Gram-Schmidt map $GS(\cdot)$ applied per token to the matrix of expert output vectors $E=[e_1,\dots,e_k]$, producing orthonormal columns $E'$ with $E'^T E'=I_k$; the set of such matrices is the Stiefel manifold $S(d,k)$. The forward pass uses $E'$ in place of $E$ in the mixture sum, so orthogonality is enforced architecturally rather than by a penalty term in the loss. The Gram-Schmidt step is what converts 'more experts' into 'diverse experts', and it is the reason the paper can claim diversity without changing the learning objective.

What would settle it

Remove the Gram-Schmidt step from OMoE and retrain on the same benchmarks with identical experts, rank, routing, and random seeds; if accuracy does not drop materially, the orthogonality constraint is not the source of the reported gains.

Watch

Extended reading notes

Core claim

OMoE's discovery is that expert diversity, not expert count, is the bottleneck in LoRA-based mixtures. Soft-routed and top-2-routed vanilla MoE both show large overlapping regions in t-SNE projections of expert representations, and performance saturates as experts are added. OMoE replaces the standard weighted sum over experts with a sum over the Gram-Schmidt-orthogonalized outputs $E' = GS(E)$, so that for each token the experts' contribution vectors are mutually orthogonal. This hard constraint is applied directly in the forward computation, leaving the supervised fine-tuning objective $J(\phi)$ unchanged. In single-task and multi-task experiments on LLaMA-2 7B/13B and LLaMA-3 8B, OMoE with two experts and soft routing consistently matches or exceeds LoRA, DoRA, and MixLoRA while cutting tunable parameters from about 2.9% of backbone size to about 0.73%, a roughly 75% reduction.

Load-bearing premise

The load-bearing premise is that the per-token Gram-Schmidt orthogonalization is a stable, differentiable operation that keeps the information the task needs while forcing the expert outputs apart; if it discards task-relevant information or misbehaves under gradient descent, the performance claim would collapse.

Editorial extensions

If this is right

  • If the central claim holds, the default practice of scaling MoE-LoRA by adding experts is wasteful: two orthogonal experts recover or exceed the accuracy of eight non-orthogonal ones on these benchmarks.
  • OMoE obtains its gains without a router balancing loss or top-k selection, since the reported configuration uses two experts with soft routing, removing a source of tuning complexity in MoE-LoRA.
  • The constraint transfers across adapter decompositions, working on DoRA as well as LoRA, so orthogonalization is a general add-on for low-rank adapter mixtures rather than a fix for one method.
  • The roughly 75% reduction in tunable parameters at equal or better accuracy widens the range of memory-limited settings in which mixture-of-LoRA fine-tuning is feasible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves untested whether the per-token Gram-Schmidt step can be replaced by orthogonalizing the LoRA parameter matrices once per layer; if the gains come from the diversity of the learned subspace rather than of the instantaneous output vectors, the expensive per-token step could be avoided.
  • The paper's own observation that OMoE degrades when more than two experts use soft routing suggests the sweet spot at $k=2$ may reflect the optimization landscape pulling later experts back toward earlier ones; a follow-up could test whether an order-invariant orthogonalization preserves the gains for larger $k$.
  • Because the orthogonalization acts on the routed, weighted outputs, its effect depends on the router weights; a stress test with near-tied routing probabilities would show whether the diversity benefit survives when the router is least confident.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Orthogonal Mixture-of-Experts (OMoE), a variant of MoE-LoRA that applies a Gram-Schmidt orthogonalization to the expert representations in each forward pass, aiming to enforce the Stiefel manifold constraint E^T E = I and thereby promote expert diversity. The authors motivate the method with t-SNE visualizations of expert collapse in vanilla MoE, then evaluate OMoE on commonsense reasoning benchmarks using LLaMA-2 7B/13B and LLaMA-3 8B, reporting that two orthogonal experts with 0.73% tunable parameters outperform or match stronger-parameter baselines such as LoRA, DoRA, and MixLoRA. The paper also includes multi-task experiments, ablations on rank and expert count, layer-wise analyses, and efficiency measurements.

Significance. If the central mechanism is validated, OMoE would be a simple and useful contribution to PEFT: it introduces a hard architectural constraint rather than an auxiliary diversity loss, and the reported efficiency gains are appealing. The paper's strengths include a clear motivating visualization, a straightforward method, and evaluation on multiple backbones and tasks. However, the current evidence does not isolate the orthogonalization from other architectural choices, the written equation for Gram-Schmidt is incomplete, and the 'consistent improvement' claim is stronger than the tables support. The contribution is therefore plausible but not yet established to the standard needed for a journal publication.

major comments (4)
  1. [3.2, Eq. (7)] The recurrence in Eq. (7) omits the normalization step required for orthonormality: e'_1 is never normalized, and e'_k is only projected to be orthogonal to previous vectors but not rescaled to unit norm. As written, the forward pass does not guarantee E'^T E' = I, so the claimed hard constraint of Eq. (4) and the Stiefel-manifold membership are not implemented by the written algorithm. The authors should state the full Gram-Schmidt procedure with normalization or explicitly describe a separate renormalization step.
  2. [4.4] The explanation for observed redundancy when k>=3 is internally inconsistent with the method's mathematics. If Eqs. (6)-(7) produce per-token orthonormal expert outputs E' satisfying E'^T E' = I, then the representations actually fed into Eq. (5) are orthogonal; the provided vector example (v1=(1,0,0), v2=(0,0,5), v3=(0,0.9,0)) concerns pre-orthogonalization vectors and does not explain redundancy in the post-orthogonalization space. The authors should either clarify that the visualized redundancy arises from t-SNE projection or from the router weighting, or rewrite the discussion to match what the method actually computes.
  3. [4.2 and 4.3 (Tables 1-2, Fig. 4)] The central attribution claim is confounded. In Tables 1-2, OMoE always uses 2 experts, rank 16, and soft routing, while MixLoRA uses 8 experts, rank 16, and top-2 routing, and LoRA/DoRA use rank 80. The only experiment that varies the orthogonalization alone is Fig. 4, which reports only averaged curves with no error bars, seed counts, or per-task numbers. To support the claim that orthogonalization—rather than the smaller two-expert architecture—produces the gains, the authors should add a controlled comparison, for example OMoE vs. a vanilla 2-expert MoE with identical rank and routing, and MixLoRA with 2 experts.
  4. [4.2, Table 1] The abstract and Section 4 claim OMoE 'consistently' outperforms state-of-the-art baselines, but Table 1 shows notable counterexamples. For LLaMA-2 7B, OMoE-LoRA is 9.6 points below MixLoRA on Winogrande (67.2 vs. 76.8) and 1.0 point below on OBQA (80.6 vs. 81.6); several other gaps are within 1 point. Because no error bars or multiple-seed results are reported and most baseline numbers are taken from other papers with different training setups, the consistency claim is not supported by the presented evidence.
minor comments (6)
  1. [Abstract] The word 'Ulteriorly' is not standard English in this context; consider 'Furthermore' or 'Additionally'.
  2. [2.3] References to 'Fig.1(c)(d)(e)' appear mislabeled relative to the panel structure of Figure 1; the text should cite the specific panels that show the relevant comparisons.
  3. [Appendix B] 'LLaMA-2 8B' in the pretrained-backbones paragraph should read 'LLaMA-3 8B' to match the main text.
  4. [Appendix D/E captions] The captions for Figures 6-11 contain the typo 'representaion'; this should be corrected.
  5. [Table 1] The header 'LLaMA-2 13B-bf' is not explained; please define 'bf' or use the standard model name.
  6. [4.3] The statement that r=32 performance declines 'due to the difficulty in convergence under limited training steps' is not supported by any learning curves or convergence analysis; either add evidence or rephrase as a conjecture.

Circularity Check

0 steps flagged · score 0.0 of 10

OMoE is an empirical PEFT paper; its performance claims are tested on independent benchmarks and no fitted parameter is renamed as a prediction, so there is no circularity.

full rationale

No circular step was found. The central claim is empirical: OMoE applies a fixed Gram-Schmidt transform to LoRA expert outputs (Eqs. 5-7) and is evaluated on held-out commonsense benchmarks (Tables 1-3) against external baselines. No parameter is fitted to the target benchmark and then renamed as a prediction; the transform is parameter-free and the comparisons are with independent methods. The motivating observations about representation collapse are qualitative and do not enter the benchmark evaluation as fitted inputs. References to prior orthogonal-finetuning and orthogonal-MoE work are not a load-bearing self-citation chain: the method is assessed by its measured performance, not by appeal to the authors' earlier results. The inconsistency noted in Sec. 4.4, where later experts are said to become similar despite orthogonality, is a limitation of the diversity mechanism rather than evidence of circular derivation, and the omission of explicit normalization in Eq. (7) is a correctness concern, not a circularity concern. The paper's claims are therefore self-contained with respect to the benchmark evidence.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on a small set of hyperparameters (rank, number of experts, routing strategy) and on the assumption that the Gram-Schmidt operation is a well-behaved differentiable projection. No new theoretical entities are introduced.

free parameters (5)
  • LoRA rank r = 16
    Chosen based on the ablation in Table 4, where performance peaks at r=16 and declines at r=32. This is a hand-selected hyperparameter, not derived from theory.
  • LoRA alpha = 32
    Standard scaling factor in LoRA; fixed across experiments, not tuned per task.
  • Number of experts k = 2
    Chosen based on Fig. 4, where k=2 gives the best trade-off between performance and parameter count; performance degrades for larger k under soft routing.
  • Routing strategy = Soft routing
    Default in the main results; the paper claims no router training in the efficiency analysis, but the forward equation (Eq. 5) includes a router g, creating ambiguity.
  • Learning rate = 2e-4
    Fixed hyperparameter listed in Table 9; no sensitivity analysis is provided.
assumptions (4)
  • domain assumption Gram-Schmidt orthogonalization is differentiable and yields stable gradients for backpropagation.
    The paper does not analyze gradient flow through the orthogonalization process; it assumes standard automatic differentiation handles the operation correctly.
  • domain assumption The pretrained LLM representations are suitable for fine-tuning via low-rank updates.
    This is a standard assumption in the PEFT literature, but it is not explicitly proven or tested.
  • ad hoc to paper The observation of 'expert collapse' via t-SNE visualizations is the true cause of the performance plateau in MoE-LoRA.
    This is the paper's motiv ating hypothesis, supported only by qualitative visualizations and performance curves, not by a formal analysis or causal experiment.
  • domain assumption Baseline results extracted from original papers are comparable to OMoE results despite different training setups.
    The paper mixes numbers from different papers (e.g., LoRA rank 80, MixLoRA rank 16) without controlling for all training details, which could bias the comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuning." pith.science (2026). https://pith.science/paper/TR5NRPWK

@misc{pith2026250110062,
  author       = {Pith},
  title        = {Pith review of: OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TR5NRPWK}},
  note         = {Machine review of arXiv:2501.10062}
}
read the original abstract

Building mixture-of-experts (MoE) architecture for Low-rank adaptation (LoRA) is emerging as a potential direction in parameter-efficient fine-tuning (PEFT) for its modular design and remarkable performance. However, simply stacking the number of experts cannot guarantee significant improvement. In this work, we first conduct qualitative analysis to indicate that experts collapse to similar representations in vanilla MoE, limiting the capacity of modular design and computational efficiency. Ulteriorly, Our analysis reveals that the performance of previous MoE variants maybe limited by a lack of diversity among experts. Motivated by these findings, we propose Orthogonal Mixture-of-Experts (OMoE), a resource-efficient MoE variant that trains experts in an orthogonal manner to promote diversity. In OMoE, a Gram-Schmidt process is leveraged to enforce that the experts' representations lie within the Stiefel manifold. By applying orthogonal constraints directly to the architecture, OMoE keeps the learning objective unchanged, without compromising optimality. Our method is simple and alleviates memory bottlenecks, as it incurs minimal experts compared to vanilla MoE models. Experiments on diverse commonsense reasoning benchmarks demonstrate that OMoE can consistently achieve stable and efficient performance improvement when compared with the state-of-the-art methods while significantly reducing the number of required experts.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AnchorMoE: Interpretable Time Series Classification via Anchor-Routed MoE

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    AnchorMoE turns multivariate time-series classification into an additive Mixture-of-Experts over multi-view patches, with orthogonal anchors and reliability gates that suppress noise while matching strong classifiers.

Reference graph

Works this paper leans on

47 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [26]

    B. Liu, L. Ding, L. Shen, K. Peng, Y . Cao, D. Cheng, and D. Tao. Diver- sifying the mixture-of-experts representation for language models with orthogonal optimizer. arXiv preprint arXiv:2310.09762, 2023

    arXiv 2023

  2. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 tech- nical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [2]

    Y . Bisk, R. Zellers, J. Gao, Y . Choi, et al. Piqa: Reasoning about physi- cal commonsense in natural language. In Proceedings of the AAAI con- ference on artificial intelligence, volume 34, pages 7432–7439, 2020

    work page 2020

  4. [3]

    contextual

    A. Chan. Gpt-3 and instructgpt: technological dystopianism, utopi- anism, and “contextual” perspectives in ai ethics and industry. AI and Ethics, 3(1):53–64, 2023

    work page 2023

  5. [4]

    S. Chen, Z. Jie, and L. Ma. Llava-mole: Sparse mixture of lora ex- perts for mitigating data conflicts in instruction finetuning mllms.arXiv preprint arXiv:2401.16160, 2024

    arXiv 2024

  6. [5]

    Z. Chi, L. Dong, S. Huang, D. Dai, S. Ma, B. Patra, S. Singhal, P. Ba- jaj, X. Song, X.-L. Mao, et al. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Sys- tems, 35:34600–34613, 2022

    work page 2022

  7. [6]

    Clark, K

    C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    arXiv 1905

  8. [7]

    Clark, I

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

Show all 47 references
  1. [8]

    S. Dou, E. Zhou, Y . Liu, S. Gao, J. Zhao, W. Shen, Y . Zhou, Z. Xi, X. Wang, X. Fan, et al. Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment. arXiv preprint arXiv:2312.09979, 4(7), 2023

    2023 arXiv

  2. [9]

    Hayou, N

    S. Hayou, N. Ghosh, and B. Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354, 2024

    2024 arXiv

  3. [10]

    J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366, 2021

    2021 arXiv

  4. [11]

    K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026–1034, 2015

    2015

  5. [12]

    Hendawy, J

    A. Hendawy, J. Peters, and C. D’Eramo. Multi-task reinforce- ment learning with mixture of orthogonal experts. arXiv preprint arXiv:2311.11385, 2023

    2023 arXiv

  6. [13]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  7. [14]

    S. Hu, Z. Zhang, N. Ding, Y . Wang, Y . Wang, Z. Liu, and M. Sun. Sparse structure search for parameter-efficient tuning. arXiv preprint arXiv:2206.07382, 2022

    2022 arXiv

  8. [15]

    Z. Hu, L. Wang, Y . Lan, W. Xu, E.-P. Lim, L. Bing, X. Xu, S. Poria, and R. Lee. Llm-adapters: An adapter family for parameter-efficient fine- tuning of large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 5254...

    2023

  9. [16]

    Huang, Q

    C. Huang, Q. Liu, B. Y . Lin, T. Pang, C. Du, and M. Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269, 2023

    2023 arXiv

  10. [17]

    Huang, Y

    Y . Huang, Y . Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y . Zhang, Y . Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Informa- tion Processing Systems, 36, 2024

    2024

  11. [18]

    Kalajdzievski

    D. Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732, 2023

    2023 arXiv

  12. [19]

    W. Kim, Y . Shin, J. Park, and Y . Sung. Sample-efficient and safe deep reinforcement learning via reset deep ensemble agents. Advances in Neural Information Processing Systems, 36, 2024

    2024

  13. [20]

    D. J. Kopiczko, T. Blankevoort, and Y . M. Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454, 2023

    2023 arXiv

  14. [21]

    S. Lan, R. Zhang, Q. Yi, J. Guo, S. Peng, Y . Gao, F. Wu, R. Chen, Z. Du, X. Hu, et al. Contrastive modules with temporal attention for multi-task reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  15. [22]

    D. Li, Y . Ma, N. Wang, Z. Cheng, L. Duan, J. Zuo, C. Yang, and M. Tang. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. arXiv preprint arXiv:2404.15159, 2024

    2024 arXiv

  16. [23]

    J. Li, L. Fuxin, and S. Todorovic. Efficient riemannian optimiza- tion on the stiefel manifold via the cayley transform. arXiv preprint arXiv:2002.01113, 2020

    2002 arXiv

  17. [24]

    X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  18. [25]

    T. Lin, J. Liu, W. Zhang, Z. Li, Y . Dai, H. Li, Z. Yu, W. He, J. Li, H. Jiang, et al. Teamlora: Boosting low-rank adaptation with expert collaboration and competition. arXiv preprint arXiv:2408.09856, 2024

    2024 arXiv

  19. [27]

    H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965, 2022

    1950

  20. [28]

    Q. Liu, X. Wu, X. Zhao, Y . Zhu, D. Xu, F. Tian, and Y . Zheng. Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications. arXiv preprint arXiv:2310.18339, 2023

    2023 arXiv

  21. [29]

    Liu, C.-Y

    S.-Y . Liu, C.-Y . Wang, H. Yin, P. Molchanov, Y .-C. F. Wang, K.-T. Cheng, and M.-H. Chen. Dora: Weight-decomposed low-rank adap- tation. arXiv preprint arXiv:2402.09353, 2024

    2024 arXiv

  22. [30]

    X. Liu, K. Ji, Y . Fu, W. L. Tam, Z. Du, Z. Yang, and J. Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021

    2021 arXiv

  23. [31]

    Mihaylov, P

    T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  24. [32]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language mod- els to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  25. [33]

    C. Qian, Z. Xie, Y . Wang, W. Liu, Y . Dang, Z. Du, W. Chen, C. Yang, Z. Liu, and M. Sun. Scaling large-language-model-based multi-agent collaboration. arXiv preprint arXiv:2406.07155, 2024

    2024 arXiv

  26. [34]

    Z. Qiu, W. Liu, H. Feng, Y . Xue, Y . Feng, Z. Liu, D. Zhang, A. Weller, and B. Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems , 36: 79320–79362, 2023

    2023

  27. [35]

    Sakaguchi, R

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021

    2021

  28. [36]

    M. Sap, H. Rashkin, D. Chen, R. LeBras, and Y . Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019

    1904 arXiv

  29. [37]

    C. Shi, C. Yang, X. Zhu, J. Wang, T. Wu, S. Li, D. Cai, Y . Yang, and Y . Meng. Unchosen experts can contribute too: Unleashing moe mod- els’ power by self-contrast. arXiv preprint arXiv:2405.14507, 2024

    2024 arXiv

  30. [38]

    R. Song, Z. Liu, X. Chen, H. An, Z. Zhang, X. Wang, and H. Xu. Label prompt for multi-label text classification. Applied Intelligence, 53(8): 8761–8775, 2023

    2023

  31. [39]

    C. Tian, Z. Shi, Z. Guo, L. Li, and C. Xu. Hydralora: An asym- metric lora architecture for efficient fine-tuning. arXiv preprint arXiv:2404.19245, 2024

    2024 arXiv

  32. [40]

    P. Wang, H. Zheng, S. Dai, W. Yue, W. Zhu, and X. Wang. Ts-tcd: Triplet-level cross-modal distillation for time-series forecasting using large language models. arXiv preprint arXiv:2409.14978, 2024

    2024 arXiv

  33. [41]

    E. B. Zaken, S. Ravfogel, and Y . Goldberg. Bitfit: Simple parameter- efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199, 2021

    2021

  34. [42]

    Zellers, A

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi. Hel- laswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  35. [43]

    Zhang, Y

    J. Zhang, Y . Zhao, D. Chen, X. Tian, H. Zheng, and W. Zhu. Milora: Efficient mixture of low-rank adaptation for large language models fine- tuning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 17071–17084, 2024

    2024

  36. [44]

    Zhang, M

    Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y . Cheng, W. Chen, and T. Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512 , 2023

    2023 arXiv

  37. [45]

    Zhang, P

    Y . Zhang, P. Wang, M. Tan, and W. Zhu. Learned adapters are better than manually designed adapters. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7420–7437, 2023

    2023

  38. [46]

    W. Zhu, A. X. Tian, C. Yin, Y . Ni, X. Wang, and G. Xie. Iapt: Instruction-aware prompt tuning for large language models. arXiv preprint arXiv:2405.18203, 2024

    2024 arXiv

  39. [47]

    Zhuang, Y

    Y . Zhuang, Y . Yu, K. Wang, H. Sun, and C. Zhang. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems, 36:50117–50143, 2023. Datasets #train #test Type Metrics BoolQ 9,427 3,270 Text Classification acc OBQA 4,957...

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.