{"id":"5ebc148c-633d-42f8-86ca-1470811f073e","arxiv_id":"2412.10135","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ASLoRA shares the low-rank A matrix across all layers and adaptively merges similar B matrices during fine-tuning, claiming LoRA-level or better performance with roughly 75% fewer trainable parameters.","lead":"The paper proposes ASLoRA, a LoRA variant that shares the low-rank A matrix across all transformer layers and adaptively merges the layer-specific B matrices during training. It reports that ASLoRA matches or beats LoRA on GLUE and instruction-tuning benchmarks while using about a quarter of the trainable parameters.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The adaptive merging heuristic (L2 distance between averaged B matrices, Eq. 4) is not validated against random or alternative pairing; without that control, the core claim that adaptive sharing drives the gains is unsupported.","rationale":"The reader's weakest_assumption identifies precisely the load-bearing premise I would attack: the merging heuristic. The paper's experiments do not ablate the selection mechanism itself. The pre-experiment (Figure 1) varies only the number of shared-B groups and shows some sharing helps, not that the L2-selected pairs are the right ones. Table 3 compares adaptive to fixed adjacent sharing while holding parameter counts equal, but that only demonstrates that non-adjacent sharing is flexible, not that the L2 metric chooses the best non-adjacent pairs. A control using random pairing is necessary to attribute any gain to the proposed metric. This concern is sharpened by the absence of error bars and by the very small GLUE average margin (0.2 points). A secondary but concrete inconsistency: the abstract's 'less than 25% of the parameters' holds only for GLUE (24%), while the instruction-tuning results use 8.9M/33.6M = 26%, contradicting the stated bound. This does not undermine the broad contribution, but it reinforces the conditional status. The internal inconsistencies (rank r = 4 in Table 4 vs r = 64 in §4.2; undefined 'Update Ratio λ' in Table 5) are additional reasons not to accept the empirical claims at face value. My proposed random-pairing ablation would settle whether the adaptive rule itself is load-bearing; until then, CONDITIONAL is the appropriate verdict, matching the reader.","tokens_in":14142,"tokens_out":7754,"duration_ms":77794,"concrete_test":"Run ASLoRA on the same GLUE tasks with the same hyperparameters, but replace the similarity-based pairing with random pairing of B matrices at each merge step, repeated over several seeds. If random pairing performs as well as L2-based pairing on average, the adaptive criterion is not load-bearing. Additionally, compute the correlation between pairwise L2 distances of averaged B's and the measured increase in validation loss when each candidate pair is forcibly merged (by averaging both B's and measuring loss before further training); a low correlation would directly invalidate Eq. 4 as a shareability proxy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ASLoRA outperforms LoRA with fewer parameters rests on the adaptive merging rule in §3.3. The rule selects the pair of B matrices with the smallest L2 distance between their time-averaged values (Eq. 4) and forces the lower layer to adopt the upper layer's B. The paper never shows that this distance predicts the actual performance cost of merging a given pair. Figure 1 only varies the number of adjacent shared-B groups, not which pairs are shared; Table 3 compares adaptive to fixed adjacent sharing at equal parameter counts, but any flexible non-adjacent sharing pattern could produce the same improvement, so that comparison does not isolate the L2 criterion. If the L2 proxy is not informative, the proposed adaptive mechanism is arbitrary, and the reported gains could stem from generic parameter tying plus per-dataset tuning of Ts, m, and N (Table 5) rather than from identifying genuinely shareable layers. The absence of error bars and the marginal GLUE average gain (85.4 vs 85.2) make this concern acute.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ASLoRA, a parameter-efficient fine-tuning method in which the LoRA matrix A is shared across all layers while each layer initially has its own B matrix. During training, after Ts steps, the method computes the L2 distance between time-averaged B matrices (Eq. 4) every m steps, merges the two most similar B matrices by having the lower layer adopt the upper layer's B, and repeats this N times before a final optimization phase. Experiments on RoBERTa-base with GLUE and on LLaMA-2-7B with instruction-tuning benchmarks report that ASLoRA matches or improves on LoRA while using about 24% of the parameters on GLUE and about 26% on instruction tuning. Additional analyses compare adaptive merging with fixed adjacent sharing, visualize the learned sharing patterns, and study the effect of the number of merges.","tokens_in":14397,"tokens_out":11401,"duration_ms":113357,"significance":"If the empirical claims hold, ASLoRA offers a simple way to reduce per-task adapter storage by roughly 75% while retaining or slightly improving downstream quality, which is practically valuable. The paper's strengths include a clearly specified algorithmic idea, experiments on two model families, and a limitations section that acknowledges hyperparameter sensitivity. The central claim, however, rests on small average margins, an unvalidated similarity heuristic, and per-dataset tuning of hyperparameters, so the current evidence is not fully conclusive. I also note that the only self-citation (MELoRA) appears in related/future work and is not used to support the results, so there is no circularity concern.","major_comments":[{"comment":"The 'Advantage of Adaptive Sharing' comparison does not isolate the L2-distance criterion. Table 3 compares L2-based adaptive merging against fixed adjacent merging at equal parameter counts, but because the adaptive method can merge non-adjacent layers, the improvement could come from the flexibility to form non-adjacent groups rather than from the L2 proxy. To support the core mechanism, add control experiments such as random non-adjacent pairing, an alternative similarity measure (e.g., cosine similarity or activation-space distance), or an oracle that merges based on validation performance. Without such controls, the paper does not establish that the L2 criterion is what drives the reported gains.","section":"§4.3, Table 3"},{"comment":"The headline claims are stronger than the numbers support. The abstract says ASLoRA 'outperforms LoRA while using less than 25% of the parameters,' but the instruction-tuning configuration in Table 2 uses 8.9M versus 33.6M, which is 26.5%, not below 25%. On GLUE, LoRA actually achieves higher scores on 4 of 6 datasets (SST-2, CoLA, QNLI, STS-B) and the average gain is 0.2 points (85.4 vs. 85.2) with no error bars or significance tests. Section 1's claim that ASLoRA is 'outperforming the baseline models across all instruction-following datasets' is also contradicted by Table 2 for full fine-tuning on MMLU (47.30 vs. 46.21). Please report repeated-seed means and variances, check the parameter-ratio arithmetic, and qualify the claims accordingly.","section":"§4.1, Table 1, abstract, Section 1"},{"comment":"The main result is conditioned on per-dataset hyperparameter selection. Table 5 lists different start-step Ts and merge-interval W for every GLUE dataset, and Figure 4 shows that the optimal merge count varies across instruction datasets (N=24 for MMLU, N=20 for BBH, N=16 for HEval). Moreover, Section 4.2 says the maximum number of merges is 16, while Figure 4 reports settings up to 28. Please specify a fixed protocol or an explicit model-selection rule and resolve the inconsistency; otherwise the parameter-efficiency comparison is not for a single method but for per-dataset tuned configurations.","section":"§4.2, Table 5, Figure 4"},{"comment":"The pseudocode is ambiguous and internally inconsistent. Line 4 of Algorithm 1 says 'Update Bi by equation (3),' but Eq. (3) defines a running average B^t_i, not an update rule. After two B matrices are merged, it is unclear whether subsequent average-weight computations use the merged parameters or the pre-merge history. The text following Algorithm 1 also says 'we calculate the pairwise similarity between adjacent layers every m steps,' which contradicts §3.3's non-adjacent merging and the non-adjacent patterns shown in Figure 3. Please rewrite the algorithm so the exact merging procedure is reproducible.","section":"Algorithm 1 and Eq. (3)"}],"minor_comments":[{"comment":"Table 4 lists Rank r=4 for instruction tuning, while §4.2 states that all methods use r=64. The parameter counts in Table 2 imply r=64, so Table 4 appears incorrect and should be fixed.","section":"Table 4 vs. §4.2"},{"comment":"The hyperparameter 'Update Ratio λ 0.5' appears in Table 5 but λ is never defined or used in Algorithm 1 or any equation; either remove it or specify its role in the method.","section":"Table 5"},{"comment":"The label 'ASLoRA-adp' is used for the fixed-sharing baseline, which is confusing because 'adp' suggests adaptive. Please rename the baseline (e.g., 'ASLoRA-fixed') to avoid ambiguity.","section":"Table 3"},{"comment":"The text says the method merges the B matrices with 'the highest similarity' and the algorithm explanation says 'the lowest similarity'; the two statements are consistent under Eq. (4), where a smaller L2 distance means higher similarity, but this should be stated explicitly to prevent misreading.","section":"Algorithm 1, §3.3"},{"comment":"No code or configuration files are released. Given that the method introduces new hyperparameters (Ts, m, N) and the paper reports per-dataset settings, providing code would substantially aid reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper has a usable empirical idea, but the headline claims need to be reconciled with the reported numbers, and the core similarity heuristic would benefit from direct controls. The rank inconsistency between Table 4 and §4.2 should be checked carefully; it may indicate that some experimental settings were changed between versions. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate incremental extension of ShareLoRA/Tied LoRA. The new piece is adaptive merging of B matrices during training based on L2 distance between time-averaged B's, plus a three-stage schedule. The paper does what a method paper should: it shows the sharing pre-experiment, compares against fixed sharing at equal parameter counts, and reports parameter savings honestly. The limitations section is candid about the per-dataset hyperparameters.\n\nThe experiments are directionally consistent: on GLUE, ASLoRA uses 0.073M params vs LoRA's 0.3M and averages 85.4 vs 85.2; on instruction tuning it uses 8.9M vs 33.6M and beats LoRA on average, with the HEval gap being the most noticeable. That said, the magnitude is modest and there are no error bars, so part of that could be noise.\n\nThe load-bearing weakness is the one the stress-test flags: the L2 similarity rule is never validated as a predictor of merge cost. Table 3 compares adaptive merging to fixed adjacent merging, but that does not isolate the L2 criterion; any flexible non-adjacent scheme could produce the same benefit. Without a control where pairs are merged randomly or by some other cheap rule at the same parameter count, the paper's claim that the specific similarity-based selection drives the gains is unsupported. The fact that the merge schedule and count are tuned per dataset (Ts, m, N in Table 5) makes it worse, because the adaptive rule's advantage could be partly a search over schedules.\n\nThere are also small internal inconsistencies that should be fixed: the text says rank 64 for LLaMA but Appendix Table 4 says 4; Algorithm 1 says merge 'lowest similarity' while the text says highest; and lambda appears in Table 5 without ever being defined or used. None of these are fatal, but they need cleanup.\n\nWho should read this: people doing PEFT and multi-model serving who care about cutting adapter storage. The method is simple enough to be useful even if the theory behind the merge criterion is thin.\n\nRecommendation: send it to peer review. A competent referee can push for the random-pairing control and error bars, and the paper would be acceptable after those revisions. It is honest, reproducible in principle, and the core idea is not outlandish. I would not cite it in my own work yet, but I would read the revised version.","headline":"Plausible parameter-sharing trick with directionally supportive experiments, but the adaptive merging criterion is not tested against random pairing and the reported gains are small enough that the central claim is only conditional.","tokens_in":14919,"tokens_out":3238,"would_cite":false,"duration_ms":27883,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sharing low-rank layers beats LoRA at a quarter of the parameters","keywords":["parameter-efficient fine-tuning","low-rank adaptation","cross-layer parameter sharing","adaptive merging","instruction tuning","GLUE","LoRA"],"falsifier":"Run ASLoRA with the same merge budget but pairing B matrices at random instead of by L2 similarity. If random pairing matches the adaptive rule's accuracy at equal parameter counts, then the similarity heuristic is not what drives the gain. A more direct test: keep the merge schedule but replace the overwrite rule with the average of the two B matrices; if performance collapses, the specific direction of overwriting is essential.","tokens_in":13950,"feed_emoji":"🧩","tokens_out":4961,"duration_ms":44205,"temperature":0.7,"pith_summary":"The paper proposes ASLoRA, a low-rank fine-tuning method that shares the A matrix across every layer and progressively merges the layer-specific B matrices during training. The central claim is that this cross-layer sharing schedule outperforms standard LoRA while using less than 25% of trainable parameters, on both GLUE-style NLU tasks and instruction tuning of a 7B language model. The authors argue that sharing A captures global information, while adaptively merging B removes redundancy while preserving layer-specific knowledge. If true, this offers a cheap way to shrink the storage and memory footprint of fine-tuned adapters without sacrificing quality.","feed_headline":"Sharing low-rank layers beats LoRA at a quarter of the parameters","feed_subtitle":"A simple merge rule cuts adapter storage by ~75% while matching or beating full-rank fine-tuning on instruction tasks.","key_machinery":"The central mechanism is the adaptive merging rule: after a shared-training phase, each B matrix is replaced by its time-averaged value (Eq. 3), pairwise L2 distances between these averages (Eq. 4) are computed across layers, and the closest pair is merged by overwriting the lower layer's B with the upper layer's B; the process repeats every m steps until N merges. This is paired with full sharing of the rank-r matrix A across all layers, which removes A's random initialization as a confound in the similarity comparison. The merge rule is what carries the parameter reduction: with 7 of a possible 11 merges on a 12-layer model, per-layer B count drops enough to cut trainable parameters to about 24% of LoRA's.","core_discovery":"ASLoRA's core proposal is a three-stage training procedure: first, train with a single globally-shared A and per-layer B matrices; second, every m steps after step Ts, compute time-averaged B matrices, rank all pairs by L2 distance, and replace the two most similar B matrices by the upper layer's B; third, continue training with the merged, partially-shared B structure until convergence. The paper reports that on six GLUE tasks with RoBERTa-base, ASLoRA reaches an average score of 85.4 with 0.073M trainable parameters, versus LoRA's 85.2 with 0.3M parameters, and on LLaMA-2-7B instruction tuning it averages 32.33 across MMLU, BBH, DROP, and HumanEval, beating LoRA's 31.40 while using 8.9M parameters to LoRA's 33.6M. It further claims that adaptive merging outperforms fixed every-n-layer sharing at equal parameter budgets.","pith_inferences":["One natural extension the paper leaves implicit: the same merge rule could be applied to intra-layer structure, e.g., merging low-rank factors across the query and value projections, or across heads, to push parameter counts even lower.","The L2-similarity proxy is a direct, testable hypothesis about representational redundancy; a stronger test would compare it against merge rules based on gradient alignment or Fisher information, which might better preserve task-critical directions.","If the merge heuristic is validated more broadly, it suggests that much of a fine-tuned adapter's per-layer variation is redundant, and that cross-layer sharing could become a default compression step before quantization in deployment pipelines."],"forward_implications":["Adapter storage for fine-tuned LLMs can be cut by roughly 75% relative to LoRA without losing average task performance, on both encoder and decoder models.","The adaptive merge schedule yields better results than fixed every-n-layer sharing at matched parameter budgets, particularly when the number of merges is small.","The optimal number of merges is task-dependent; performance on MMLU peaks at 24 merges, on BBH at 20, and on HumanEval at 16, so the schedule can be tuned per downstream task.","ASLoRA's parameter count shrinks as model depth grows, since more layers allow more merges; the method's advantage widens with model size."],"supporting_citations":[{"why":"Defines the LoRA adapter (B and A low-rank matrices) that ASLoRA modifies and the baseline it must beat.","marker":"(Hu et al., 2022)"},{"why":"ShareLoRA, the direct predecessor that shares A across layers while leaving B unshared; ASLoRA extends this to adaptive sharing of B.","marker":"(Song et al., 2024)"},{"why":"VeRA, a sharing-based PEFT baseline that freezes shared random A and B and trains scaling vectors; ASLoRA contrasts its own trainable sharing with this approach.","marker":"(Kopiczko et al., 2024)"},{"why":"AdaLoRA, a baseline for comparison and the source of the claim that upper layers encode more complex information, used to justify the overwrite direction.","marker":"(Zhang et al., 2023b)"},{"why":"Tied LoRA, a weight-tying baseline whose binding mechanism lacks the flexibility of ASLoRA's adaptive merging.","marker":"(Renduchintala et al., 2024)"},{"why":"InstructEval, the evaluation protocol for the instruction-tuning experiments on MMLU, BBH, DROP, and HumanEval.","marker":"(Chia et al., 2024)"}],"fun_headline_variants":["Adaptive sharing cuts LoRA params to one-quarter, improves scores","ASLoRA: Share one A, adaptively merge B's, quarter the params","Cross-layer low-rank sharing outperforms LoRA at 1/4 cost","Adaptive LoRA sharing beats full LoRA with 75% fewer params"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the L2 distance between time-averaged B matrices identifies which layers can safely share a B matrix, and that overwriting the lower layer's B with the upper layer's B preserves the useful information in that pair.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive sharing cuts LoRA params to one-quarter, improves scores","ASLoRA: Share one A, adaptively merge B's, quarter the params","Cross-layer low-rank sharing outperforms LoRA at 1/4 cost","Adaptive LoRA sharing beats full LoRA with 75% fewer params"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00075,"raw_usage":{"total_tokens":3325,"prompt_tokens":919,"completion_tokens":2406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":2323}},"tokens_in":535,"tokens_out":2406,"duration_ms":18222,"temperature":1.0,"reasoning_tokens":2323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:18:25.006602+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ASLoRA with the same merge budget but pairing B matrices at random instead of by L2 similarity. If random pairing matches the adaptive rule's accuracy at equal parameter counts, then the similarity heuristic is not what drives the gain. A more direct test: keep the merge schedule but replace the overwrite rule with the average of the two B matrices; if performance collapses, the specific direction of overwriting is essential.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VeRA, a sharing-based PEFT baseline that freezes shared random A and B and trains scaling vectors; ASLoRA contrasts its own trainable sharing with this approach."}],"review_version":1}