{"id":"5faf1013-0479-4ef0-8f29-e5d495c2fa6f","arxiv_id":"2412.13488","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A static sparsity mask chosen by plain gradients matches or beats LoRA and second-order salience metrics across NLP fine-tuning benchmarks.","lead":"The paper compares eight ways to choose which weights to fine-tune in a large language model using sparse adapters, and finds a simple gradient-based rule works as well or better than fancier alternatives. It also tests whether updating the selection during training helps, and concludes that a fixed mask chosen before training is just as good, which could simplify efficient fine-tuning in practice.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Comparison with LoRA/PiSSA may be confounded: Appendix A Table 6 uses 10x lower learning rates for sparse SPEFT than for LoRA on the same MMLU datasets (e.g., 5e-6 vs 5e-5 on OASST2), contradicting the stated hyperparameter-fairness claim and leaving the large-model margin unresolved.","rationale":"The reader's weakest_assumption is the same one I would put first: the headline comparison is vulnerable to unequal hyperparameter treatment. My reading of Appendix A confirms the contradiction: Table 6 reports different learning rates for SPEFT and LoRA on the same datasets, despite Section 4's claim that all remaining hyperparameters were kept the same. I therefore agree with the CONDITIONAL verdict. The proposed test is deliberately narrow: only the MMLU rows in Table 2 are affected, but those rows are the main large-model evidence for \"consistently outperforms\"; their margins (about 0.5 and 0.1 points) are small enough that a 10x learning-rate change could plausibly flip them. I did not elevate the BERT dynamic-masking result (Table 4: DG 88.27 vs SG 87.42) to the central concern, because that would challenge the secondary static-vs-dynamic claim rather than the primary comparison against LoRA/PiSSA. The paper does provide a public repository and three-seed repeats for small models, which is real evidence; the concern is about experimental control, not about the authors' honesty. If the symmetric sweep shows gradient SPEFT still ahead at its optimum, the paper's central claim would be substantially strengthened and the CONDITIONAL verdict could be upgraded; if not, the claim should be moderated to \"competitive on MMLU, better on math/code.\"","tokens_in":17495,"tokens_out":8896,"duration_ms":79829,"concrete_test":"Rerun the two confounded MMLU configurations (Gemma2-2b on OASST2, Qwen2-7b on OASST2) with a symmetric learning-rate sweep for LoRA, PiSSA, and gradient SPEFT over {1e-6, 5e-6, 1e-5, 5e-5, 1e-4}, using the Appendix A settings otherwise and 3 seeds per configuration. Compare each method at its own best learning rate. If best-tuned LoRA or PiSSA reaches or beats gradient SPEFT on MMLU, the \"consistently outperforms\" claim fails for the large-model benchmark; if gradient SPEFT still wins under best-vs-best, the fairness concern is resolved and the original margin stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the central comparison is not yet shown to be apples-to-apples. Section 4 states that \"kept the remaining hyperparameters always the same\" when comparing SPEFT with LoRA/PiSSA, but Appendix A does not: Table 6 gives Gemma2-2b a sparse SPEFT learning rate of 1e-5 on Alpaca and 5e-6 on OASST2 while LoRA uses 5e-5 on both; for Qwen2-7b, SPEFT uses 5e-6 on OASST2 while LoRA uses 5e-5. These are 10x differences on exactly the MMLU rows where the reported advantage is small (e.g., Gemma2-2b OASST2: 53.11 vs 52.59; Qwen2-7b OASST2: 70.55 vs 70.42). If the low-rank baselines were simply run at a less favorable learning rate for those datasets, the claimed \"consistently outperforms\" could shrink or reverse. The concern is not that a per-method learning rate is illegitimate; it is that the paper neither acknowledges the difference nor documents a symmetric best-vs-best tuning procedure for the larger models, so the reader cannot tell whether the margin is a method effect or a learning-rate artifact. Notably, the learning-rate mismatch is confined to Table 2/Table 6; the GLUE, MetaMathQA, and CodeFeedback tables use shared learning rates, so the concern is specifically about the MMLU evidence, which is the main large-model support for the headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies sparsity-based parameter-efficient fine-tuning (SPEFT), where a sparse additive weight matrix is fine-tuned while pretrained weights are frozen. It evaluates eight salience metrics (magnitude, gradient, SNIP, FORCE, Taylor-FO, SynFlow, GRaSP, Fisher information) for constructing the initial sparsity mask, compares static versus dynamically updated masks and global versus local sparsity, and benchmarks against LoRA and PiSSA on GLUE (OPT, BERT, RoBERTa), MMLU (Gemma2-2b, Qwen2-7b), GSM8K (Gemma2-2b on MetaMathQA), and HumanEval/MBPP (Llama3-8b on CodeFeedback). The main claim is that a simple gradient-based, static SPEFT variant consistently outperforms other PEFT methods and is a strong baseline. The paper also argues that dynamic masking provides no substantial benefit over static masking. It releases open-source code.","tokens_in":17837,"tokens_out":6715,"duration_ms":56213,"significance":"If the central claim held, the contribution would be practically valuable: a static, gradient-magnitude mask selected once at initialization gives a cheaper PEFT alternative that matches or exceeds low-rank adapters on several benchmarks. The study's systematic comparison of salience proxies, the inclusion of both static and dynamic masking, and the release of code are strengths, as are the repeated runs with standard deviations for the smaller models. However, the headline claim of consistent superiority is not fully supported by the paper's own tables, and the large-model comparison contains a learning-rate asymmetry that must be resolved. The paper's contribution is real but needs refinement.","major_comments":[{"comment":"The claim that 'kept the remaining hyperparameters always the same' (Section 4) is contradicted by Table 6, where Gemma2-2b and Qwen2-7b use SPEFT learning rates of 1e-5 or 5e-6 while LoRA uses 5e-5 on the same Alpaca/OASST2 setups. These 10x differences occur on the exact MMLU rows in Table 2 that support the large-model advantage, with margins of only 0.04–1.19 points (e.g., Gemma2-2b OASST2: 53.11 vs 52.59; Qwen2-7b OASST2: 70.55 vs 70.42). Because no symmetric best-vs-best tuning procedure is documented for these larger models, I cannot tell whether the reported gain is a method effect or a learning-rate artifact. The concern is not that per-method learning rates differ; it is that the paper neither acknowledges the difference nor shows that the comparison is apples-to-apples. This must be resolved before the central 'consistently outperforms' claim is acceptable.","section":"§4, Appendix A Table 6"},{"comment":"In the code-generation experiments, the data do not support the abstract's claim that gradient-based SPEFT 'consistently outperforms other fine-tuning methods.' Taylor-FO achieves a higher average (49.40 vs 49.39) and a higher HumanEval score (49.39 vs 48.78), while SNIP is close (48.98). Section 4.2's statement that 'the lead by sparse adapters widens' is therefore unsupported on HumanEval/MBPP. The claim should be restricted to the benchmarks where it holds, or qualified with statistical significance tests.","section":"Table 12"},{"comment":"The word 'consistently' is also too strong on GLUE. While gradient-based SPEFT has the best average on several models, Fisher-Info wins more tasks on BERT-base (Table 1: 3 vs 2) and PiSSA wins more on RoBERTa-base (Table 9: 3 vs 2). The sentence 'gradient-based SPEFT has the best average accuracy, higher than LoRA and PiSSA' is fine, but the introduction and abstract overstate per-task consistency. Please temper the language or add a paired-test analysis.","section":"Tables 1, 9, 10"}],"minor_comments":[{"comment":"'Results are on par with the best alternatives' is difficult to reconcile with the later sentence 'consistently outperforms other fine-tuning methods'; please choose one or state the qualification.","section":"Abstract"},{"comment":"The FORCE row reports an average of 34.91, which is inconsistent with HumanEval 46.95 and MBPP 50.4 (the average should be about 48.68); this appears to be a typo and should be corrected.","section":"Table 12"},{"comment":"Table 2 does not include the PiSSA baseline, although Section 4 states that PiSSA is included as a low-rank baseline across models; please clarify whether PiSSA was run on the large models or why it is omitted.","section":"Table 2"},{"comment":"The statement that salience estimation is 'negligible' refers to a one-time cost for static masks; for dynamic masks the cost is incurred every I steps. Please make this distinction explicit in the efficiency discussion.","section":"Section 4.4"},{"comment":"The admission that SPEFT appeared less sensitive to hyperparameters than LoRA was not explored is important; given the learning-rate differences in Table 6, this unexplored observation should at least be mentioned in the main text as a caveat.","section":"Section 8"}],"recommendation":"major_revision","confidential_remarks":"This is a potentially useful empirical study, but the headline claim is stronger than the evidence. The learning-rate asymmetry in Appendix A Table 6 is the critical issue; I recommend requesting a revised version that either equalizes tuning effort or documents a systematic best-of-N search for both families, and that tempers claims of consistency to match the per-task results. The paper is within scope for the journal and, after revisions, could be a solid empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is the systematic comparison: eight salience metrics, static vs dynamic masks, across six model families and several task types. That benchmark is worth having, and the main practical message—simple gradient-based static SPEFT is a strong, cheap baseline—is persuasive. The study is large and mostly careful: repeated runs for smaller models, ablations on masking strategy, overhead measurements, and released code.\n\nThe soft spots are real but not fatal. First, the abstract says gradient-based SPEFT \"consistently outperforms\" other methods, but the paper's own tables undercut that. Fisher-Info wins three of six GLUE tasks on BERT-base, Taylor-FO matches or beats gradient on code (Table 12), and GRaSP is close on GSM8K. \"Consistently\" is doing too much work; the data support \"competitive and often best.\"\n\nMore serious is the hyperparameter mismatch the stress-test flags. In Appendix A Table 6, on the same MMLU datasets, SPEFT uses 1e-5 or 5e-6 while LoRA uses 5e-5—a tenfold difference—even though Section 4 claims \"kept the remaining hyperparameters always the same.\" This affects exactly the rows where the reported margin is a fraction of a point (e.g., Gemma2-2b OASST2: 53.11 vs 52.59). That is a legitimate confound, and the authors do not acknowledge it or document a symmetric best-vs-best tuning procedure for the larger models. Notably, the GLUE, MetaMathQA, and CodeFeedback tables share learning rates, so the core comparison partially holds; the problem is specifically the MMLU evidence that carries the headline claim. Also, large-model results are single runs, so sub-point differences are within noise.\n\nWhat the paper does well: it assembles a useful public benchmark, tests static vs dynamic masking and global vs local sparsity, and honestly notes in the limitations that SPEFT seemed less sensitive to hyperparameters than LoRA—an observation worth following up.\n\nWho is this for? Anyone doing PEFT and wanting a baseline that is simpler than LoRA and often matches it. It deserves a serious referee, but I would send it back for major revision: retune the low-rank baselines with a symmetric sweep or report a range over learning rates, and soften the consistency claim to match the tables. The MMLU comparison needs to be re-run or explicitly caveated. If that is fixed, this becomes a solid reference point for the field.","headline":"Useful broad benchmark of salience metrics for sparse PEFT, but the headline claim of consistent superiority over LoRA is weakened by a learning-rate mismatch in the MMLU experiments.","tokens_in":18418,"tokens_out":1971,"would_cite":true,"duration_ms":18326,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A static mask chosen once by gradient magnitude is the best way to pick sparse fine-tuning weights, beating LoRA and PiSSA across NLP benchmarks.","keywords":["parameter-efficient fine-tuning","sparse fine-tuning","salience metrics","static masks","dynamic masks","LoRA","large language models","gradient-based selection"],"falsifier":"Re-run the GLUE, MMLU, and GSM8K comparisons with the best learning rate for each method found by the same sweep protocol; if LoRA or PiSSA then matches or beats gradient-based static SPEFT on average, the paper's central claim would collapse.","tokens_in":17274,"feed_emoji":"🧠","tokens_out":6849,"duration_ms":58249,"temperature":0.7,"pith_summary":"The paper asks which salience metric should choose the few trainable weights in sparsity-based parameter-efficient fine-tuning (SPEFT), and whether the chosen mask should stay fixed or be refreshed during training. It evaluates eight first- and second-order salience metrics across GLUE, MMLU, GSM8K, and code benchmarks, and finds that plain first-order gradient information is the most reliable selector. It also finds that a static mask computed once before training performs as well as dynamic re-masking, at a fraction of the compute. The paper claims that a simple gradient-based, static SPEFT variant consistently outperforms LoRA and PiSSA when all methods use the same number of trainable parameters, and proposes it as a strong baseline for PEFT.","feed_headline":"Static gradient masks beat LoRA and PiSSA","feed_subtitle":"Per-weight gradients chosen once outperform dynamic masks and low-rank baselines on GLUE, MMLU, math, and code.","key_machinery":"The central object is the sparse adapter $\\theta_{\\mathrm{sp}}$ with binary mask $\\tau \\in \\{0,1\\}^{d_1 \\times d_2}$, reparameterizing each layer as $\\theta = \\theta_0 + \\theta_{\\mathrm{sp}}$ with $1[\\theta_{\\mathrm{sp}} \\neq 0] = \\tau$ and $|\\tau|_0 \\le \\rho$. The mask is built by computing one of eight salience scores $S(\\theta)$ and keeping the top $\\rho$ fraction of values, either globally across all layers or locally within each layer. The mechanism that carries the argument is the gradient salience score $\\partial \\ell/\\partial \\theta$ combined with a static mask: it requires only a short pre-training gradient estimation (64 steps of batch size 16), and during fine-tuning only the masked nonzero entries of $\\theta_{\\mathrm{sp}}$ are updated. This zero-cost selection and the absence of mask updates are what make the method both simple and efficient.","core_discovery":"The paper's central discovery is that, in sparse PEFT, where a frozen pretrained weight matrix $\\theta_0$ is augmented by a trainable sparse matrix $\\theta_{\\mathrm{sp}}$ whose nonzero entries are fixed by a mask $\\tau$, the simple gradient $\\partial \\ell/\\partial \\theta$ is the most reliable salience score for constructing the mask. It outperforms magnitude, SNIP, FORCE, Taylor-FO, SynFlow, GRaSP, and Fisher information on the evaluated tasks. The paper further argues that static masking, which selects the top-$\\rho$ gradient values once before training and never changes them, is sufficient: dynamic masking refreshed every 1000 steps offers no substantial benefit and adds computational overhead from re-ranking and optimizer reinitialization. On this basis, the paper claims that gradient-based static SPEFT consistently beats LoRA and PiSSA on GLUE, MMLU, GSM8K, HumanEval, and MBPP with matched trainable-parameter counts.","pith_inferences":["The paper leaves open whether the same ordering of salience metrics holds when sparse adapters are combined with other efficiency techniques, such as quantization of the frozen base model; a natural extension would freeze the gradient-chosen mask and quantize the residual $\\theta_{\\mathrm{sp}}$.","If the result transfers beyond language models, it suggests that parameter location, not low-rank structure, is what makes PEFT work; this could be tested by comparing gradient-selected sparse adapters against full-rank adapters with the same number of trainable parameters.","The paper's finding that global and local sparsity give comparable results hints that per-layer budget allocation matters less than the choice of salience signal, which could be probed with adaptive per-layer budgets.","Since the static mask is chosen before training, it could be combined with mixtures of sparse experts or multi-task sparse mask transfer, as the paper notes, to reuse one mask across related tasks."],"forward_implications":["A static gradient-based sparse adapter can serve as a simple default PEFT baseline, offering a cheaper alternative to dynamic masking and a competitive or better alternative to LoRA and PiSSA at equal parameter budgets.","Second-order salience metrics such as Fisher information and GRaSP do not pay for themselves in SPEFT, so first-order gradient information should be the starting point for mask construction.","Dynamic masking's extra computational cost, from periodic mask recomputation and optimizer reinitialization, is not justified by its empirical performance.","The advantage of gradient-based SPEFT appears to widen on harder generation and reasoning tasks, such as GSM8K math word problems and code generation, where it beats LoRA by large margins.","As hardware support for sparse matrix multiplication matures, the efficiency advantage of static SPEFT should grow, since the fixed mask is known in advance and can be compiled into sparse kernels."],"supporting_citations":[{"why":"Defines the LoRA low-rank reparameterization and serves as the primary baseline that gradient-based static SPEFT is compared against.","marker":"(Hu et al., 2021)"},{"why":"Provides the competing low-rank baseline PiSSA, based on principal singular values and vectors, used throughout the experiments.","marker":"(Meng et al., 2024)"},{"why":"Defines SNIP connection sensitivity, one of the gradient-based salience metrics evaluated in the study.","marker":"(Lee et al., 2019a)"},{"why":"Introduces static Fisher-information masks, the prior static SPEFT approach that this work builds on and compares to other mask strategies.","marker":"(Sung et al., 2021)"},{"why":"Defines FORCE, a foresight connection sensitivity metric that is among the eight salience metrics evaluated.","marker":"(de Jorge et al., 2021)"},{"why":"Defines SynFlow, the data-free salience metric tested as a mask-construction proxy.","marker":"(Tanaka et al., 2020)"},{"why":"Defines Taylor-FO, a first-order expansion-based salience metric included in the comparison.","marker":"(Molchanov et al., 2019)"},{"why":"Defines GRaSP, a second-order gradient-signal-preservation metric included in the comparison.","marker":"(Wang et al., 2020)"}],"fun_headline_variants":["Simple gradient-based masks outperform LoRA and PiSSA","Sparse fine-tuning: static gradient masks win","Gradient salience beats dynamic masks for PEFT","Static gradients: simple and strongest for sparse PEFT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison depends on LoRA and PiSSA having received learning-rate tuning as careful as the sparse adapters, but Appendix A shows different learning rates for the two families on the same datasets, so a less-tuned baseline could explain part of the reported gap.","fun_headline_variants_meta":{"raw":{"variants":["Simple gradient-based masks outperform LoRA and PiSSA","Sparse fine-tuning: static gradient masks win","Gradient salience beats dynamic masks for PEFT","Static gradients: simple and strongest for sparse PEFT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1604,"prompt_tokens":953,"completion_tokens":651,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":588}},"tokens_in":569,"tokens_out":651,"duration_ms":6401,"temperature":1.0,"reasoning_tokens":588,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:04:46.906155+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the GLUE, MMLU, and GSM8K comparisons with the best learning rate for each method found by the same sweep protocol; if LoRA or PiSSA then matches or beats gradient-based static SPEFT on average, the paper's central claim would collapse.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces static Fisher-information masks, the prior static SPEFT approach that this work builds on and compares to other mask strategies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines FORCE, a foresight connection sensitivity metric that is among the eight salience metrics evaluated."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines SynFlow, the data-free salience metric tested as a mask-construction proxy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Taylor-FO, a first-order expansion-based salience metric included in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines GRaSP, a second-order gradient-signal-preservation metric included in the comparison."}],"review_version":1}