MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning

Dacao Zhang; Kun Zhang; Le Wu; Shimao Chu; Si Wei; Xin Li

REVIEW 3 major objections 8 minor 1 cited by

MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning

T0 review · 3 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MoRE claims a single LoRA module can serve many tasks by treating each rank as an expert and selecting the rank per task with a learned gating function, improving multi-task performance.

desk verdict Original rank-as-expert idea with broad experiments, but Eq. 7 and the contrastive loss have sign/semantics errors that must be resolved before the results are reproducible. read the letter →

arxiv 2505.22694 v1 pith:DGBKXHSP submitted 2025-05-28 cs.LG

Dacao Zhang , Kun Zhang , Shimao Chu , Le Wu , Xin Li , Si Wei This is my paper · ORCID

classification cs.LG

keywords mixtureofexpertslow-rankadaptation(LoRA)multi-tasklearningparameter-efficientfine-tuningtaskembeddingscontrastiveadaptiverankselectionGLUEbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes MoRE, a parameter-efficient fine-tuning method for multi-task LLM adaptation. Its central claim is that instead of training one LoRA per task or a set of parallel LoRA experts, a single LoRA module can treat each rank as an 'expert' and let a learned task embedding pick the rank that suits each task. On GLUE, commonsense reasoning, and few-shot transfer experiments, the authors report that MoRE outperforms LoRA, MultiLoRA, MixLoRA, and MOELoRA with roughly the same parameter count as a single LoRA, and the abstract asserts there is no additional inference cost. The significance, if true, is that multi-task PEFT can be adaptive without paying the parameter and routing overhead of parallel experts.

What carries the argument

The rank-as-expert construction: one LoRA module of maximum rank $r$, where the selector can pick any $r_t \in [1,r]$ and use the shared prefix of $A$ and $B$. The gating network maps a task embedding to a rank distribution, the straight-through estimator makes the discrete selection trainable, and a linear scaling by $r_t/|T|$ stabilizes the frequently-used shared lower ranks. Task embeddings are tuned by a contrastive loss so that similar tasks cluster, and balanced dataset sampling mixes the task data. This machinery is what allows a single LoRA to give each task a different capacity while sharing parameters across tasks.

What would settle it

Inspect the released implementation of the sampling distribution and benchmark per-query latency of the fixed task-to-rank mode against LoRA $r=8$: if $\phi_t$ in Eq. (7) is applied as written, dataset weights grow with size rather than inversely, so the balanced-sampling ablation (86.2 vs 87.3) does not test the stated mechanism, and if latency is flat the abstract's 'no additional inference cost' holds, otherwise the limitations section's latency admission does.

Watch

Extended reading notes

Core claim

MoRE treats each rank of a single LoRA module as a low-rank expert. For a task with learned embedding $e_t$, a gating network $G(e_t) = \mathrm{softmax}(W_g e_t + b_g)$ scores ranks $1,\dots,r$, and the forward pass truncates the LoRA matrices to the selected rank $r_t$ via $A_t = A[:r_t,:]$, $B_t = B[:,:r_t]$; a straight-through estimator lets the discrete selection receive gradients. Task embeddings are learned with a contrastive loss that keeps each task's samples close to its embedding, and a balanced dataset sampling strategy mixes the task data during training. The paper reports that on GLUE, MoRE reaches 87.3 average (T5-base) and 88.8 (LLaMA2-7B), surpassing LoRA and parallel-LoRA baselines like MultiLoRA, MixLoRA, and MOELoRA; it also reports stronger few-shot domain transfer and a parameter count comparable to a single LoRA when the trained task-to-rank mapping is cached.

Load-bearing premise

The method's experiments rest on the description of the balanced sampling strategy in Section 4.3, but the text says the weights are inversely proportional to dataset size while Eq. (7) defines weights that grow with dataset size, so if the implementation follows the equation the reported mechanism and the ablation attributed to it do not hold as described; separately, the abstract's 'no additional inference cost' is contradicted by the limitations section's admission that the MoE structure cannot be merged and adds latency.

Editorial extensions

If this is right

Multi-task fine-tuning can be run with the parameter budget of a single LoRA while outperforming parallel-LoRA designs on GLUE and commonsense reasoning.
Task similarity is encoded by embedding proximity and therefore rank proximity, which enables few-shot transfer by initializing a new task's embedding near a similar task's.
After training, the adaptive selector can be replaced with a cached task-to-rank mapping, removing the gate from the per-sample forward pass.
Single-task dynamic-rank methods like DyLoRA can be extended to multi-task by conditioning rank choice on task identity through the learned embeddings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The abstract's 'no additional inference cost' conflicts with the limitations section's statement that the MoE structure cannot be merged and adds latency, so the no-cost claim should be read as parameter-count saving, not wall-clock saving.
If Eq. (7) is implemented as written, the 'balanced' sampling weights grow with dataset size, making the 86.2-vs-87.3 ablation result attributable to a mechanism opposite to the one described; the released code can settle this.
The rank-as-expert scheme could be ported to other PEFT structures, with 'expert' meaning an adapter block or prefix length instead of a rank, using the same task-embedding gate.
If task embeddings cluster by task family as the visualizations suggest, MoRE provides a cheap automatic measure of task similarity usable for task grouping or curriculum design.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read. The core idea—treating the rank slices of a single LoRA module as a mixture of experts and routing tasks to them with a learned task embedding—is genuinely new relative to MultiLoRA, MixLoRA, and MoELoRA, which all use parallel LoRA modules. That reformulation is the paper's real contribution, and it's a reasonable one. The experiments are broad: GLUE, commonsense reasoning, few-shot transfer, and NLG, on T5-base and LLaMA2-7B, with parameter counts and training-speed analysis. The gains are modest—about one GLUE point average over strong baselines—but they are consistent, and the code is released. That's a legitimate incremental contribution.

The soft spots are real, though. Section 4.3 defines the balanced sampling weight as phi_t = exp(|D_t|/sum|D_i|), which grows with dataset size, while the text says the weight is inversely proportional to size. Those cannot both be right. The ablation in Table 5 attributing a 1.1-point gain to balanced sampling is therefore uninterpretable as written. If the code actually uses inverse-size weights, the equation is a typo; if it follows the equation, the method upweights MNLI and downweights RTE, which is the opposite of balancing. Either way, the paper needs to show the actual formula or the code.

Second, the contrastive loss in Eq. 8 is written as a positive log-probability and then added to the generation loss and minimized. Minimizing a negative log-probability as a loss would drive the probability down; the usual InfoNCE form has a minus sign. That looks like a dropped sign, but as written it's backwards.

Third, the abstract's 'no additional inference cost' is contradicted by the Limitations section's admission of latency from the MoE structure. Appendix B's task-to-expert mapping removes the routing computation but still leaves an unmerged low-rank update to apply, so the claim needs to be scoped carefully.

Finally, the main tables lack error bars. With differences around one point, we need to know whether any of this is noise. The few-shot experiments report five seeds, so they know how to do it; the main results should too.

This is not a desk-reject. The idea is sound and the experimental design is thoughtful. But the equation-level contradictions and missing variance reporting need to be fixed before the results can be trusted. A serious referee should see it, with the expectation of a careful revision.

Referee Report

3 major / 8 minor

Summary. The paper proposes MoRE, a parameter-efficient fine-tuning method for multi-task LLM adaptation. The core idea is to treat each rank of a single LoRA module as a 'low-rank expert' and use a gating network over learned task embeddings to select, for each task, the appropriate rank; the selected rank truncates the shared LoRA matrices, so lower ranks are shared across tasks while higher ranks are used only by some tasks. The method also adds a contrastive-learning objective for the task embeddings and a so-called balanced dataset sampling strategy. Experiments on GLUE, commonsense reasoning, few-shot transfer, and NLG with T5-base and LLaMA2-7B report average improvements over several multi-task LoRA baselines with competitive parameter counts.

Significance. If the claims hold, the nested-rank-expert design is a novel and plausible way to share parameters across tasks without training multiple LoRA modules, and the reported improvements are practically relevant. The paper provides a code release and a clear ablation structure. However, the manuscript currently contains two major internal inconsistencies—the sampling equation in Section 4.3 and the inference-cost claim—that must be resolved before the results can be interpreted as stated.

major comments (3)

[Section 4.3, Eq. (7)] The sampling weight φ_t = exp(|D_t|/Σ_i |D_i|) is monotonically increasing in |D_t|, yet the text states it is 'inversely proportional to its size.' As written, the scheme upweights large datasets such as MNLI and downweights small ones such as RTE, which is the opposite of balancing, and the expression is not normalized for use as a sampling distribution. This is load-bearing because the Table 5 ablation attributes a 1.1-point GLUE improvement to 'balanced' sampling; if the implementation actually followed Eq. (7), that improvement cannot be attributed to the claimed mechanism. The authors must correct the equation or the surrounding text and clarify which sampling procedure was used in the experiments.
[Abstract and Section 7] The abstract claims that MoRE 'incurs no additional inference cost,' but Section 7 states that because the approach is based on the MoE structure, it 'cannot be merged with the original model, it results in latency during inference.' Appendix B asserts that a task-to-expert mapping allows the router and task embeddings to be bypassed at inference, reducing the parameter count to that of LoRA. These statements are contradictory. The authors should specify the exact inference procedure and reconcile the abstract with the Limitations section.
[Tables 2 and 3] The headline claim of 'significantly improves' is not supported by any measure of variance. For example, in Table 3 MoRE's average of 82.7 is only 0.7 points above MixLoRA's 81.9 and 0.7 above MOELoRA's 82.0, and in Table 2 several individual-task differences are within a few tenths. The few-shot experiments (Table 4) report 5-seed averages but no standard deviations. Without error bars or significance tests, the 'significant' improvement claim cannot be assessed.

minor comments (8)

[Section 5 heading] The heading 'Expertments' is a typo and should read 'Experiments.'
[Eq. (5)] The expression Ste(pt)[rt] is confusing because, in the forward pass, Ste(pt) is a one-hot vector, so indexing by rt always yields 1; the notation should be clarified to show that the scalar multiply is 1 in the forward direction and pt[rt] in the backward direction.
[Eq. (6)] The linear scaling rt/|T| is described as making the learning rate 'small' for the shared lower part, but the scaling is applied to the forward output, not to the optimizer's learning rate; the intended effect should be stated precisely.
[Eq. (8)] The representation h_i used in the contrastive loss is not defined precisely; the paper should state which layer or module's output is used for each sample.
[Section 4.1 and Table 6] Section 4.1 says the method uses a single task embedding matrix E, but the parameter formula 6Lh(r+T) in Table 6 suggests each LoRA module has its own task embeddings; clarify whether task embeddings are shared across layers and LoRA modules.
[Table 6 and Appendix B] The parameter formula for MoRE, 6Lr(m+d) + 6Lh(r+T), combined with h=768 and T=r=8, does not obviously yield the 0.78M value reported in Table 2; the parameter accounting should be made consistent.
[Appendix B] The text contains typos: 'matraix' should be 'matrix' and 'paramters' should be 'parameters.'
[Table 5] The 'w/ Subset Experts' ablation is not fully defined; the paper should specify how the subset experts are formed and selected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the method is end-to-end trained and no prediction is defined in terms of its own fitted outputs.

full rationale

The paper's derivation chain is self-contained supervised training. The three core components, task embeddings (Section 4.1), the adaptive rank selector (Eqs. 2-6), and the contrastive objective (Eq. 8), are all learned parameters optimized jointly with the generation loss (Eq. 9); no fitted value is subsequently presented as an independent prediction. The rank selector is trained end-to-end with a straight-through estimator, so its output is a function of learned embeddings and gate weights, not a restatement of the benchmark numbers on which it is evaluated. The motivating rank sweep in Table 1 reports per-task LoRA performance, but MoRE does not use those best ranks as supervision; the selector learns its own allocation, and the paper only analyzes the resulting allocation after training. Citations to LoRA, DyLoRA, AdaLoRA, and SoRA are standard prior work and are not used to justify a load-bearing premise about MoRE. There is no uniqueness theorem, no fitted parameter renamed as a prediction, and no equation in which the predicted quantity is defined in terms of the evaluation target. Two non-circular issues should be noted as correctness risks rather than circularity: (i) Section 4.3, Eq. (7) defines phi_t = exp(|D_t| / sum_i |D_i|), which increases with dataset size, while the surrounding text says the sampling weight is 'inversely proportional to its size', making the balanced-sampling ablation not reproducible as written; and (ii) the abstract claims 'no additional inference cost', while Section 7 states that the MoE structure 'results in latency during inference', a tension that Appendix B's task-to-expert mapping may partially resolve but that the paper should state precisely. Neither issue makes a result equivalent to its inputs by construction, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The method rests on standard supervised training without any derivation from first principles. The main load-bearing components are the learned task embeddings, the routing mechanism, and the heuristic balanced sampling. None have independent external evidence beyond the paper's own experiments, and one component (balanced sampling) is described inconsistently in the text.

free parameters (5)

lambda = 0.1
Hyperparameter balancing generation and contrastive losses; chosen from the sensitivity test in Figure 4.
tau = 0.05
Softmax temperature in the contrastive loss, chosen by hand without a sensitivity analysis.
task_embedding_dim = 768
Selected via sensitivity analysis (Figure 4) as the best among tested dimensions.
max_rank_r = 8
Maximum rank for the single LoRA module; matches LoRA baseline settings, but the paper does not study its effect in MoRE.
linear_scaling_factor = rt/|T|
Heuristic multiplier intended to slow updates of the shared lower ranks; introduced ad hoc and only ablated as a whole.

assumptions (4)

domain assumption Different tasks have different optimal LoRA ranks.
Motivated by Table 1, which shows best ranks varying across MRPC and CoLA, but not established generally for all multi-task settings.
domain assumption The model representation hi can be aligned with the task embedding et of the sample's task.
The paper asserts this without proof; the contrastive loss is built on this comparability, and the exact layer for hi is unspecified.
standard math Straight-through Estimator gives a useful gradient for the discrete rank selection.
STE is a known technique, but its bias for this discrete selection is not analyzed in the paper.
ad hoc to paper Linear scaling by rt/|T| stabilizes training of the shared lower-rank part.
Introduced heuristically and only ablated as a whole; no theoretical justification is provided.

invented entities (2)

Task embedding
purpose: Learned vector per task that feeds the rank selector and contrastive loss.
No external validation; quality only inferred from downstream performance and PCA visualizations.
Rank expert
purpose: A rank slice of the LoRA module treated as a task-specific expert.
Re-interpretation of existing LoRA parameters; its validity is the paper's central claim rather than an independently verified entity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning." pith.science (2026). https://pith.science/paper/DGBKXHSP

@misc{pith2026250522694,
  author       = {Pith},
  title        = {Pith review of: MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DGBKXHSP}},
  note         = {Machine review of arXiv:2505.22694}
}

read the original abstract

With the rapid development of Large Language Models (LLMs), Parameter-Efficient Fine-Tuning (PEFT) methods have gained significant attention, which aims to achieve efficient fine-tuning of LLMs with fewer parameters. As a representative PEFT method, Low-Rank Adaptation (LoRA) introduces low-rank matrices to approximate the incremental tuning parameters and achieves impressive performance over multiple scenarios. After that, plenty of improvements have been proposed for further improvement. However, these methods either focus on single-task scenarios or separately train multiple LoRA modules for multi-task scenarios, limiting the efficiency and effectiveness of LoRA in multi-task scenarios. To better adapt to multi-task fine-tuning, in this paper, we propose a novel Mixture of Low-Rank Experts (MoRE) for multi-task PEFT. Specifically, instead of using an individual LoRA for each task, we align different ranks of LoRA module with different tasks, which we named low-rank experts. Moreover, we design a novel adaptive rank selector to select the appropriate expert for each task. By jointly training low-rank experts, MoRE can enhance the adaptability and efficiency of LoRA in multi-task scenarios. Finally, we conduct extensive experiments over multiple multi-task benchmarks along with different LLMs to verify model performance. Experimental results demonstrate that compared to traditional LoRA and its variants, MoRE significantly improves the performance of LLMs in multi-task scenarios and incurs no additional inference cost. We also release the model and code to facilitate the community.

Figures

Figures reproduced from arXiv: 2505.22694 by the authors.

**Figure 2.** (a)-(b) The distribution of expert allocation. (c) Visualization of the task embeddings. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Relative Training Speed of Different Parameter-Efficient Fine-Tuning Methods. 0.2 0.4 0.6 0.8 1.0 86.0 86.2 86.4 86.6 86.8 87.0 ACC ACC with varying W w/o Contrastive Learning 250 500 750 1000 1250 1500 hidden dim 85.8 86.0 86.2 86.4 86.6 86.8 87.0 ACC with varying hidden dim [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Parameter Sensitivity Test on λ in Eq.(10) and hidden dimension of task embedding. differentiate and link tasks, enhancing expert selection and the overall performance of MoRE. We also provide more examples in Appendix C. Training Speed Analysis [PITH_FULL_IMAGE:figu…

**Figure 5.** Figure 5: Visualization of Task Embeddings in Layer 1. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

**Figure 6.** Figure 6: Visualization of Task Embeddings in Layer 6. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

**Figure 7.** Figure 7: Visualization of Task Embeddings in Layer 12. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

ScatterPrism: convergence for generative simulation and inverse problems in particle and nuclear physics
cs.LG 2026-04 unverdicted novelty 4.0 of 10

Standard Conditional Flow Matching loss is a misleading early plateau; physics-informed metrics keep improving, so ScatterPrism and multi-metric diagnostics are needed for kinematic fidelity.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages · cited by 1 Pith paper

[9]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu

Can a suit of armor conduct elec- tricity? a new dataset for open book question answer- ing.arXiv preprint arXiv:1809.02789. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu

arXiv
[10]

Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi

Llama: Open and effi- cient foundation language models.arXiv preprint arXiv:2302.13971. Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi

arXiv
[11]

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao

Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language- models.arXiv preprint arXiv:2106.10199. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023a. Adaptive budget allocation for parameter-efficient fine-tuning. InInternational Con- ference on Learning Representations. Ope...

arXiv
[12]

to evaluate the model performance. GLUE covers multiple tasks of paraphrase de- tection (MRPC, QQP), sentiment classification (SST-2), natural language inference (MNLI, RTE, QNLI), and linguistic acceptability (CoLA). Fol- lowing previous work (Zhang et al., 2021), for those datasets with fewer than10,000 samples (i.e., RTE, MRPC, STS-B, CoLA), we split t...

work page 2021
[13]

These datasets provide a variety of challenges that require understanding of everyday scenarios and logical reasoning

datasets to assess the model’s performance in commonsense reasoning tasks. These datasets provide a variety of challenges that require understanding of everyday scenarios and logical reasoning. Moreover, we se- lect SciTail (Khot et al., 2018), BoolQ (Clark et al., 2019), and CB (de Marneffe et al.,

work page 2018
[2013]

arXiv preprint arXiv:1308.3432

Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al

arXiv
[2018]

Marie-Catherine de Marneffe, Mandy Simons, and Ju- dith Tonhauser

Think you have solved question an- swering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457. Marie-Catherine de Marneffe, Mandy Simons, and Ju- dith Tonhauser

arXiv
[2019]

Boolq: Exploring the surprising difficulty of natural yes/no questions. InProceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoen...

work page 2019

Show all 13 references

[2020]

InProceedings of the 34th International Conference on Neural Information Processing Sys- tems, pages 1877–1901

Language models are few-shot learners. InProceedings of the 34th International Conference on Neural Information Processing Sys- tems, pages 1877–1901. Shijie Chen, Yu Zhang, and Qiang Yang

1901
[2021]

InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059

The power of scale for parameter-efficient prompt tuning. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059. Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang

2021
[2022]

InProceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 6655–6672

Attempt: Parameter- efficient multi-task tuning via attentional mixtures of soft prompts. InProceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 6655–6672. Yoshua Bengio, Nicholas Léonard, and Aaron Courville

2022
[2023]

InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4133–4145

Sparse low-rank adaptation of pre-trained language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4133–4145. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mo...

2023
[2024]

arXiv preprint arXiv:2404.15159

Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. arXiv preprint arXiv:2404.15159. Xiang Lisa Li and Percy Liang

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [9]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu

Can a suit of armor conduct elec- tricity? a new dataset for open book question answer- ing.arXiv preprint arXiv:1809.02789. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu

arXiv

[2] [10]

Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi

Llama: Open and effi- cient foundation language models.arXiv preprint arXiv:2302.13971. Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi

arXiv

[3] [11]

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao

Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language- models.arXiv preprint arXiv:2106.10199. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023a. Adaptive budget allocation for parameter-efficient fine-tuning. InInternational Con- ference on Learning Representations. Ope...

arXiv

[4] [12]

to evaluate the model performance. GLUE covers multiple tasks of paraphrase de- tection (MRPC, QQP), sentiment classification (SST-2), natural language inference (MNLI, RTE, QNLI), and linguistic acceptability (CoLA). Fol- lowing previous work (Zhang et al., 2021), for those datasets with fewer than10,000 samples (i.e., RTE, MRPC, STS-B, CoLA), we split t...

work page 2021

[5] [13]

These datasets provide a variety of challenges that require understanding of everyday scenarios and logical reasoning

datasets to assess the model’s performance in commonsense reasoning tasks. These datasets provide a variety of challenges that require understanding of everyday scenarios and logical reasoning. Moreover, we se- lect SciTail (Khot et al., 2018), BoolQ (Clark et al., 2019), and CB (de Marneffe et al.,

work page 2018

[6] [2013]

arXiv preprint arXiv:1308.3432

Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al

arXiv

[7] [2018]

Marie-Catherine de Marneffe, Mandy Simons, and Ju- dith Tonhauser

Think you have solved question an- swering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457. Marie-Catherine de Marneffe, Mandy Simons, and Ju- dith Tonhauser

arXiv

[8] [2019]

Boolq: Exploring the surprising difficulty of natural yes/no questions. InProceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoen...

work page 2019