{"id":"f2b7ff09-cf5e-4fd1-979e-f7438902e548","arxiv_id":"2505.22694","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MoRE trains one LoRA module whose different rank slices act as per-task experts, selected by a task-embedding router, improving multi-task fine-tuning accuracy over LoRA baselines.","lead":"The paper introduces MoRE, a way to fine-tune large language models on many tasks at once by treating each rank in a single low-rank adapter as an expert and letting a learned selector pick the best rank for each task. It claims better multi-task accuracy than standard LoRA and its variants while keeping parameter counts close to a single LoRA module.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (7) contradicts its stated purpose: the sampling weights grow with dataset size, so the 'balanced sampling' component and its ablation are not reproducible as written; this should be resolved before conditional acceptance.","rationale":"The reader's weakest-assumption analysis correctly identifies the Eq. (7) / text contradiction as the most concrete and testable flaw in the paper's experimental narrative. The claimed benefit of balanced sampling is directly tied to a defined formula, and that formula does the opposite of what the text promises. This is not merely a wording issue: if the implementation follows the equation, then the mechanism being ablated is not the mechanism claimed, so the 1.1-point drop in the ablation cannot be cited as evidence for balancing. The concern is significant enough to keep the verdict conditional, but it is fixable by checking the released code and correcting either the equation or the description, so it does not warrant rejection. The inference-cost contradiction is secondary: the task-to-expert mapping described in Section 5.4 / Appendix B can support the 'no additional inference cost' claim for fixed task inference, but the Limitations section should be reconciled to avoid an apparent self-contradiction. My recommendation is unchanged conditional acceptance pending this verification.","tokens_in":14694,"tokens_out":12199,"duration_ms":136286,"concrete_test":"Inspect the released repository's data-sampling implementation. If weights are computed as exp(+|D_t|/sum|D_i|) and task selection is proportional to those weights, the paper's balanced-sampling description is false. Then re-run the GLUE experiment with the stated inverse weights (e.g., phi_t proportional to exp(-|D_t|/sum|D_i|) or to 1/|D_t|) and compare the 87.3 average. If the inverse-weight variant matches the reported average and the size-proportional variant does not, Eq. (7) is a typo and the mechanism is validated; if not, the ablation result supporting balanced sampling is unsupported and the method description needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that MoRE improves multi-task performance through adaptive rank selection plus balanced dataset sampling. In Section 4.3, Eq. (7) defines phi_t = exp(|D_t| / sum_i |D_i|), while the surrounding text says the weight is 'inversely proportional to its size.' The equation is monotonically increasing in |D_t|, so it upweights large datasets such as MNLI and downweights small ones such as RTE, which is the opposite of balancing. The ablation in Table 5 ('w/ Random Sample' 86.2 vs. 87.3) is therefore interpretable only if the implementation actually uses inverse- or size-normalized weights. If the code follows Eq. (7), the reported benefit is not attributable to balanced sampling, and the method as written cannot be reimplemented by a reader. Since the paper explicitly lists balanced data sampling as a core contribution and credits it with a performance gap, this inconsistency is load-bearing for the reproducibility of the main experimental result. A secondary but related conflict is the abstract's 'no additional inference cost' versus the Limitations section's statement that the MoE structure 'results in latency during inference'; the task-to-expert mapping in Appendix B may resolve this, but it should be stated precisely.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MoRE, a parameter-efficient fine-tuning method for multi-task LLM adaptation. The core idea is to treat each rank of a single LoRA module as a 'low-rank expert' and use a gating network over learned task embeddings to select, for each task, the appropriate rank; the selected rank truncates the shared LoRA matrices, so lower ranks are shared across tasks while higher ranks are used only by some tasks. The method also adds a contrastive-learning objective for the task embeddings and a so-called balanced dataset sampling strategy. Experiments on GLUE, commonsense reasoning, few-shot transfer, and NLG with T5-base and LLaMA2-7B report average improvements over several multi-task LoRA baselines with competitive parameter counts.","tokens_in":14989,"tokens_out":13492,"duration_ms":141931,"significance":"If the claims hold, the nested-rank-expert design is a novel and plausible way to share parameters across tasks without training multiple LoRA modules, and the reported improvements are practically relevant. The paper provides a code release and a clear ablation structure. However, the manuscript currently contains two major internal inconsistencies—the sampling equation in Section 4.3 and the inference-cost claim—that must be resolved before the results can be interpreted as stated.","major_comments":[{"comment":"The sampling weight φ_t = exp(|D_t|/Σ_i |D_i|) is monotonically increasing in |D_t|, yet the text states it is 'inversely proportional to its size.' As written, the scheme upweights large datasets such as MNLI and downweights small ones such as RTE, which is the opposite of balancing, and the expression is not normalized for use as a sampling distribution. This is load-bearing because the Table 5 ablation attributes a 1.1-point GLUE improvement to 'balanced' sampling; if the implementation actually followed Eq. (7), that improvement cannot be attributed to the claimed mechanism. The authors must correct the equation or the surrounding text and clarify which sampling procedure was used in the experiments.","section":"Section 4.3, Eq. (7)"},{"comment":"The abstract claims that MoRE 'incurs no additional inference cost,' but Section 7 states that because the approach is based on the MoE structure, it 'cannot be merged with the original model, it results in latency during inference.' Appendix B asserts that a task-to-expert mapping allows the router and task embeddings to be bypassed at inference, reducing the parameter count to that of LoRA. These statements are contradictory. The authors should specify the exact inference procedure and reconcile the abstract with the Limitations section.","section":"Abstract and Section 7"},{"comment":"The headline claim of 'significantly improves' is not supported by any measure of variance. For example, in Table 3 MoRE's average of 82.7 is only 0.7 points above MixLoRA's 81.9 and 0.7 above MOELoRA's 82.0, and in Table 2 several individual-task differences are within a few tenths. The few-shot experiments (Table 4) report 5-seed averages but no standard deviations. Without error bars or significance tests, the 'significant' improvement claim cannot be assessed.","section":"Tables 2 and 3"}],"minor_comments":[{"comment":"The heading 'Expertments' is a typo and should read 'Experiments.'","section":"Section 5 heading"},{"comment":"The expression Ste(pt)[rt] is confusing because, in the forward pass, Ste(pt) is a one-hot vector, so indexing by rt always yields 1; the notation should be clarified to show that the scalar multiply is 1 in the forward direction and pt[rt] in the backward direction.","section":"Eq. (5)"},{"comment":"The linear scaling rt/|T| is described as making the learning rate 'small' for the shared lower part, but the scaling is applied to the forward output, not to the optimizer's learning rate; the intended effect should be stated precisely.","section":"Eq. (6)"},{"comment":"The representation h_i used in the contrastive loss is not defined precisely; the paper should state which layer or module's output is used for each sample.","section":"Eq. (8)"},{"comment":"Section 4.1 says the method uses a single task embedding matrix E, but the parameter formula 6Lh(r+T) in Table 6 suggests each LoRA module has its own task embeddings; clarify whether task embeddings are shared across layers and LoRA modules.","section":"Section 4.1 and Table 6"},{"comment":"The parameter formula for MoRE, 6Lr(m+d) + 6Lh(r+T), combined with h=768 and T=r=8, does not obviously yield the 0.78M value reported in Table 2; the parameter accounting should be made consistent.","section":"Table 6 and Appendix B"},{"comment":"The text contains typos: 'matraix' should be 'matrix' and 'paramters' should be 'parameters.'","section":"Appendix B"},{"comment":"The 'w/ Subset Experts' ablation is not fully defined; the paper should specify how the subset experts are formed and selected.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The core idea has merit and the reported results are generally positive, but the two inconsistencies highlighted in the major comments are fixable in revision. In particular, I encourage the editors to ask the authors to release the exact sampling implementation and to report variance or significance measures for the main tables."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The core idea—treating the rank slices of a single LoRA module as a mixture of experts and routing tasks to them with a learned task embedding—is genuinely new relative to MultiLoRA, MixLoRA, and MoELoRA, which all use parallel LoRA modules. That reformulation is the paper's real contribution, and it's a reasonable one. The experiments are broad: GLUE, commonsense reasoning, few-shot transfer, and NLG, on T5-base and LLaMA2-7B, with parameter counts and training-speed analysis. The gains are modest—about one GLUE point average over strong baselines—but they are consistent, and the code is released. That's a legitimate incremental contribution.\n\nThe soft spots are real, though. Section 4.3 defines the balanced sampling weight as phi_t = exp(|D_t|/sum|D_i|), which grows with dataset size, while the text says the weight is inversely proportional to size. Those cannot both be right. The ablation in Table 5 attributing a 1.1-point gain to balanced sampling is therefore uninterpretable as written. If the code actually uses inverse-size weights, the equation is a typo; if it follows the equation, the method upweights MNLI and downweights RTE, which is the opposite of balancing. Either way, the paper needs to show the actual formula or the code.\n\nSecond, the contrastive loss in Eq. 8 is written as a positive log-probability and then added to the generation loss and minimized. Minimizing a negative log-probability as a loss would drive the probability down; the usual InfoNCE form has a minus sign. That looks like a dropped sign, but as written it's backwards.\n\nThird, the abstract's 'no additional inference cost' is contradicted by the Limitations section's admission of latency from the MoE structure. Appendix B's task-to-expert mapping removes the routing computation but still leaves an unmerged low-rank update to apply, so the claim needs to be scoped carefully.\n\nFinally, the main tables lack error bars. With differences around one point, we need to know whether any of this is noise. The few-shot experiments report five seeds, so they know how to do it; the main results should too.\n\nThis is not a desk-reject. The idea is sound and the experimental design is thoughtful. But the equation-level contradictions and missing variance reporting need to be fixed before the results can be trusted. A serious referee should see it, with the expectation of a careful revision.","headline":"Original rank-as-expert idea with broad experiments, but Eq. 7 and the contrastive loss have sign/semantics errors that must be resolved before the results are reproducible.","tokens_in":15491,"tokens_out":5744,"would_cite":false,"duration_ms":55714,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MoRE claims a single LoRA module can serve many tasks by treating each rank as an expert and selecting the rank per task with a learned gating function, improving multi-task performance.","keywords":["mixture of experts","low-rank adaptation (LoRA)","multi-task learning","parameter-efficient fine-tuning","task embeddings","contrastive learning","adaptive rank selection","GLUE benchmark"],"falsifier":"Inspect the released implementation of the sampling distribution and benchmark per-query latency of the fixed task-to-rank mode against LoRA $r=8$: if $\\phi_t$ in Eq. (7) is applied as written, dataset weights grow with size rather than inversely, so the balanced-sampling ablation (86.2 vs 87.3) does not test the stated mechanism, and if latency is flat the abstract's 'no additional inference cost' holds, otherwise the limitations section's latency admission does.","tokens_in":14527,"feed_emoji":"🧩","tokens_out":12750,"duration_ms":114782,"temperature":0.7,"pith_summary":"The paper proposes MoRE, a parameter-efficient fine-tuning method for multi-task LLM adaptation. Its central claim is that instead of training one LoRA per task or a set of parallel LoRA experts, a single LoRA module can treat each rank as an 'expert' and let a learned task embedding pick the rank that suits each task. On GLUE, commonsense reasoning, and few-shot transfer experiments, the authors report that MoRE outperforms LoRA, MultiLoRA, MixLoRA, and MOELoRA with roughly the same parameter count as a single LoRA, and the abstract asserts there is no additional inference cost. The significance, if true, is that multi-task PEFT can be adaptive without paying the parameter and routing overhead of parallel experts.","feed_headline":"Ranks as experts: one LoRA beats multi-LoRA on GLUE","feed_subtitle":"MoRE claims a single low-rank module adapts per task by picking its rank, with LoRA-level parameters.","key_machinery":"The rank-as-expert construction: one LoRA module of maximum rank $r$, where the selector can pick any $r_t \\in [1,r]$ and use the shared prefix of $A$ and $B$. The gating network maps a task embedding to a rank distribution, the straight-through estimator makes the discrete selection trainable, and a linear scaling by $r_t/|T|$ stabilizes the frequently-used shared lower ranks. Task embeddings are tuned by a contrastive loss so that similar tasks cluster, and balanced dataset sampling mixes the task data. This machinery is what allows a single LoRA to give each task a different capacity while sharing parameters across tasks.","core_discovery":"MoRE treats each rank of a single LoRA module as a low-rank expert. For a task with learned embedding $e_t$, a gating network $G(e_t) = \\mathrm{softmax}(W_g e_t + b_g)$ scores ranks $1,\\dots,r$, and the forward pass truncates the LoRA matrices to the selected rank $r_t$ via $A_t = A[:r_t,:]$, $B_t = B[:,:r_t]$; a straight-through estimator lets the discrete selection receive gradients. Task embeddings are learned with a contrastive loss that keeps each task's samples close to its embedding, and a balanced dataset sampling strategy mixes the task data during training. The paper reports that on GLUE, MoRE reaches 87.3 average (T5-base) and 88.8 (LLaMA2-7B), surpassing LoRA and parallel-LoRA baselines like MultiLoRA, MixLoRA, and MOELoRA; it also reports stronger few-shot domain transfer and a parameter count comparable to a single LoRA when the trained task-to-rank mapping is cached.","pith_inferences":["The abstract's 'no additional inference cost' conflicts with the limitations section's statement that the MoE structure cannot be merged and adds latency, so the no-cost claim should be read as parameter-count saving, not wall-clock saving.","If Eq. (7) is implemented as written, the 'balanced' sampling weights grow with dataset size, making the 86.2-vs-87.3 ablation result attributable to a mechanism opposite to the one described; the released code can settle this.","The rank-as-expert scheme could be ported to other PEFT structures, with 'expert' meaning an adapter block or prefix length instead of a rank, using the same task-embedding gate.","If task embeddings cluster by task family as the visualizations suggest, MoRE provides a cheap automatic measure of task similarity usable for task grouping or curriculum design."],"forward_implications":["Multi-task fine-tuning can be run with the parameter budget of a single LoRA while outperforming parallel-LoRA designs on GLUE and commonsense reasoning.","Task similarity is encoded by embedding proximity and therefore rank proximity, which enables few-shot transfer by initializing a new task's embedding near a similar task's.","After training, the adaptive selector can be replaced with a cached task-to-rank mapping, removing the gate from the per-sample forward pass.","Single-task dynamic-rank methods like DyLoRA can be extended to multi-task by conditioning rank choice on task identity through the learned embeddings."],"supporting_citations":[{"why":"Defines LoRA, the low-rank update that MoRE re-purposes into per-rank experts.","marker":"(Hu et al., 2022)"},{"why":"MultiLoRA, the parallel-LoRA baseline whose parameter count and GLUE scores MoRE compares against.","marker":"(Wang et al., 2023a)"},{"why":"MixLoRA, the gated mixture-of-LoRA baseline MoRE aims to outperform with fewer parameters.","marker":"(Li et al., 2024)"},{"why":"MOELoRA, the MoE-with-LoRA baseline that also uses task embeddings and gates.","marker":"(Liu et al., 2023a)"},{"why":"DyLoRA, the dynamic-rank training method MoRE extends to multi-task settings.","marker":"(Valipour et al., 2023)"},{"why":"Supplies the straight-through estimator used to train the discrete rank selector.","marker":"(Bengio et al., 2013)"},{"why":"GLUE, the main evaluation benchmark for the reported multi-task results.","marker":"(Wang et al., 2018)"},{"why":"HyperFormer, a hypernetwork multi-task PEFT baseline that MoRE compares against.","marker":"(Mahabadi et al., 2021)"}],"fun_headline_variants":["MoRE: one LoRA, per-task rank picks, no extra inference cost","Single LoRA outdoes multi-LoRA by choosing its rank per task","MoRE: low-rank experts that share one module, adapt per task","MoRE: task-aware rank selection boosts multi-task LoRA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's experiments rest on the description of the balanced sampling strategy in Section 4.3, but the text says the weights are inversely proportional to dataset size while Eq. (7) defines weights that grow with dataset size, so if the implementation follows the equation the reported mechanism and the ablation attributed to it do not hold as described; separately, the abstract's 'no additional inference cost' is contradicted by the limitations section's admission that the MoE structure cannot be merged and adds latency.","fun_headline_variants_meta":{"raw":{"variants":["MoRE: one LoRA, per-task rank picks, no extra inference cost","Single LoRA outdoes multi-LoRA by choosing its rank per task","MoRE: low-rank experts that share one module, adapt per task","MoRE: task-aware rank selection boosts multi-task LoRA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000521,"raw_usage":{"total_tokens":2574,"prompt_tokens":1049,"completion_tokens":1525,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":1445}},"tokens_in":665,"tokens_out":1525,"duration_ms":11262,"temperature":1.0,"reasoning_tokens":1445,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:10:31.382131+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the released implementation of the sampling distribution and benchmark per-query latency of the fixed task-to-rank mode against LoRA $r=8$: if $\\phi_t$ in Eq. (7) is applied as written, dataset weights grow with size rather than inversely, so the balanced-sampling ablation (86.2 vs 87.3) does not test the stated mechanism, and if latency is flat the abstract's 'no additional inference cost' holds, otherwise the limitations section's latency admission does.","supporting_citations":[],"review_version":1}