{"id":"e870e73e-d9cc-4ad8-a39d-84e093956e00","arxiv_id":"2506.16787","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SeLoRA reparameterizes LoRA updates as inverse Fourier or wavelet transforms of sparsely masked spectral coefficients, improving fine-tuning accuracy on LLaMA models with fewer trainable parameters.","lead":"This paper proposes SeLoRA, a way to fine-tune large language models by learning only a sparse set of spectral components instead of all low-rank adapter parameters. It reports accuracy gains over standard LoRA and related methods on reasoning and code benchmarks while using fewer trainable parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains are single-run point estimates under one fixed random spectral mask Omega; with margins of only +0.6 to +2.4 points, mask and seed sensitivity are untested, and no code or seeds are provided.","rationale":"The reader's conditional verdict already identifies the weakest assumption as the reality and generality of the sparsity property, which is threatened by single-seed artifacts, chosen eta values, and the fixed random index set Omega. My reading agrees: the central claim is empirically plausible but currently supported only by single-run point estimates with no variance information, and the lack of Omega sensitivity analysis is a genuine hole for a method whose only structural difference from LoRA is a randomly chosen spectral mask. I also independently noticed a concrete technical issue in Equation 4: retaining only the real part of the inverse DFT of a real sparse spectrum makes the Fourier reparameterization rank-deficient, because frequencies (u,v) and (r-u,d-v) map to identical cosine basis vectors. This does not invalidate the wavelet-based results, which are the better-performing variants, but it strengthens the need for a careful, reproducible evaluation before the 'spectral encoding helps' claim is accepted. The appropriate verdict remains conditional rather than accept or reject: the method is coherent and the experiments are consistent, but reproducibility and sensitivity checks are required before the reported superiority can be treated as established.","tokens_in":21105,"tokens_out":9456,"duration_ms":117434,"concrete_test":"Release the training/evaluation code and rerun the LLaMA3-8B commonsense-reasoning setup of Table 1 with at least five independent random seeds and five independently sampled Omega sets for SeLoRAW, using the same hyperparameters and eta=0.6. For each replicate, also run a coordinate-masked LoRA at the same density and with the same seed as a direct control. Report mean and standard deviation of the 8-dataset average accuracy for each method. If the mean SeLoRAW minus masked-LoRA gap is smaller than the standard deviation across replicates, the claimed advantage over LoRA cannot be attributed to spectral encoding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim, that SeLoRA beats LoRA, DoRA, and HiRA by roughly 0.6 to 2.4 points on multi-dataset averages, rests on a single optimization run and a single random draw of the shared spectral index set Omega defined in Section 2.2. Because Omega is 'randomly initialized' and fixed, every reported SeLoRA result is one sample from a distribution over masks. The paper reports no seed variance, no Omega sensitivity, and no code, so it is impossible to tell whether the observed improvements reflect a real property of spectral encoding or a favorable draw of Omega combined with usual run-to-run noise. The motivating 'sparsity property' in Figure 1 is itself presented as point curves with no error bars. A separate mathematical concern affects the Fourier variant: in Equation 4, with real-valued F_A, Re[F^{-1}(F_A)] reduces to a cosine expansion in which spectral indices (u,v) and (r-u, d-v) generate identical spatial basis vectors; the map from spectral parameters to spatial matrices is therefore not injective, and the effective number of independent Fourier degrees of freedom can be well below the stated parameter count. The wavelet variant avoids this rank collapse and the authors' own limitations section concedes that gains vanish at high rank, so the issue is not that the method is broken, but that the headline superiority is currently underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies parameter redundancy in LoRA fine-tuning, reports empirical evidence that random density masking of LoRA parameters preserves accuracy (the 'sparsity property'), and proposes SeLoRA, a reparameterization that learns only a sparse set of spectral coefficients and maps them back to the spatial domain via inverse Fourier or inverse wavelet transforms. The method is evaluated on LLaMA-2-7B and LLaMA-3-8B across commonsense reasoning, mathematical reasoning, and code generation, and is also plugged into DoRA and HiRA. The reported results show consistent accuracy gains of roughly 0.6 to 2.4 points over LoRA, DoRA, and HiRA at reduced parameter counts, together with ablations on sparsity ratio, rank, training data scale, module placement, wavelet basis, and training time.","tokens_in":21362,"tokens_out":3789,"duration_ms":44875,"significance":"If the central quantitative claim is robust, SeLoRA is a practically valuable contribution: it is simple, model-agnostic, compatible with existing LoRA variants, and shows consistent improvements at lower parameter budgets across multiple tasks and backbones. The paper is also strong in breadth: it includes careful comparisons with multiple baselines, analyses of wavelet bases, module sensitivity, rank scaling, data scaling, and efficiency, plus an honest limitations section. However, the headline improvements rest entirely on single-run point estimates with no reported variance and no released code, the shared random spectral mask is fixed for all experiments, and the Fourier variant has an unanalyzed parameter-redundancy issue. These concerns currently underdetermine the central claim, although the wavelet variant and the overall framework are plausible and worth revising rather than discarding.","major_comments":[{"comment":"All reported accuracies are single-run point estimates without standard deviations, seed counts, or confidence intervals, and no code is provided. The headline margins are typically +0.6 to +2.4 points, which is within the range of run-to-run variation commonly observed in instruction tuning of 7B–8B models. This is load-bearing because the paper's central claim is that SeLoRA consistently outperforms LoRA, DoRA, and HiRA. The motivating 'sparsity property' in Figure 1 is likewise presented as point curves with no error bars, so the claim that masking up to 60% of parameters preserves accuracy is not statistically supported. The authors should report multiple seeds with mean and variance, and ideally release code and seeds.","section":"§3.1, Tables 1–2, Figure 1"},{"comment":"The index set Ω is randomly initialized, fixed, and shared across all modules, and every SeLoRA result is obtained from a single random draw of Ω. The paper provides no sensitivity analysis over Ω, so it is unknown whether the observed gains reflect a robust property of spectral encoding or a favorable mask draw. Since Ω directly controls which spectral locations are learnable and the method's performance is comparable to baseline margins, the authors should either (a) show results across multiple independent Ω samples and report variance, or (b) provide evidence of mask invariance, such as a sweep of differently seeded masks.","section":"§2.2, Eq. (3), Section 3"},{"comment":"For real-valued spectral matrices FA, the real part of the inverse DFT is invariant under the simultaneous index exchange (u,v) ↔ (r−u, d−v), because the corresponding complex exponentials are complex conjugates and contribute identically to Re[F^{-1}(FA)]. Consequently, the linear map from spectral parameters to spatial matrices is non-injective and the reported parameter count for SeLoRAF overstates the number of independent degrees of freedom, potentially by a factor close to two (along the axes the pairing is slightly different). This matters for the parameter-efficiency claims made for the Fourier variant in Tables 1 and 2. The wavelet variant in Eq. (5)–(6) does not have this collapse, but the Fourier variant needs an explicit discussion, a corrected effective-parameter accounting, or a reformulation in terms of Hermitian-symmetric coefficients.","section":"§2.2, Eq. (4)"},{"comment":"The sparse ratio η is set separately for each task, model, and variant (e.g., 0.4 and 0.6 for commonsense, 0.2 and 0.4 for math and code), but the paper does not describe a validation protocol or held-out model selection for η. If η is chosen by evaluating on the same test benchmarks, the reported improvements may include selection bias. The authors should specify how η was selected, report the validation data used, or show that the qualitative conclusions are insensitive to η over a reasonable interval.","section":"§3.1 and §4"}],"minor_comments":[{"comment":"Typo: 'alphaca-chat prompt template' should be 'Alpaca-chat prompt template'.","section":"§3.1"},{"comment":"Typo: 'Peformance variations' should be 'Performance variations'.","section":"Table 3 caption"},{"comment":"The caption reads 'Sparse Ratioη'; there should be a space between 'Ratio' and 'η'.","section":"Figure 1 caption"},{"comment":"HiRA is implemented by the authors because the official code is not public. The description says this is 'based on the optimal configurations reported in their original papers,' but the paper should report the exact hyperparameters used for the HiRA implementation and note any adjustments, since this baseline is critical to the comparison and was not independently reproduced.","section":"§3.1, Baselines"},{"comment":"The full rank and data-scale results are useful, but they are presented without any uncertainty estimates; adding even a small number of repeated runs would strengthen the rank-scalability and data-scalability conclusions.","section":"Tables 8–9"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SeLoRA is a clean, plausible extension of LoRA: reparameterize the two low-rank factors as inverse spectral transforms of a sparse set of learned coefficients, with a fixed random mask shared across modules. The wavelet variant consistently beats LoRA, DoRA, and HiRA by roughly 1-2 points on commonsense, math, and code benchmarks while using fewer trainable parameters. That is a useful result for practitioners, and the paper does a lot right: the experiments cover two model families, three task families, and several LoRA variants, and the authors include a limitations section that openly admits the gains shrink at high rank.\n\nMy main reservation is that every reported number is a single run. No seeds, no error bars, no code. The paper also fixes one random mask Omega and never tests sensitivity to that draw. With margins of 0.6 to 2.4 points, run-to-run noise could plausibly account for part of the gap. The sparse ratio eta is tuned per task and model with no described validation protocol, so there is room for selection bias. The authors also implement HiRA themselves, which is fine, but it adds another uncontrolled variable.\n\nOn the math: the Fourier variant has a real degeneracy. Because they take the real part of the inverse DFT, spectral indices (u,v) and (r-u, d-v) generate the same spatial basis vector, so the effective number of independent Fourier coefficients is smaller than the parameter count suggests. That doesn't break the method — the wavelet variant avoids it — but it undercuts the efficiency claim for SeLoRA_F and suggests the paper should either use a cosine basis with the redundancy removed or lead with the wavelet version. The authors' own high-rank experiments confirm the gains vanish, consistent with the idea that spectral encoding mostly helps squeeze more out of a low-rank budget.\n\nOverall, the central idea is sound and the evaluation is broad, but the headline superiority is underdetermined until we see seeds, error bars, and ideally code. I'd send it to review — a good referee could push for Omega sensitivity and multi-seed runs. I wouldn't desk-reject. If the robustness checks come out okay, this could be a solid contribution to the PEFT toolbox.","headline":"Useful PEFT method with honest limitations, but single-run results and an untested random mask leave the headline gains underdetermined.","tokens_in":21890,"tokens_out":2031,"would_cite":true,"duration_ms":21643,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LoRA tolerates 60% parameter masking at fixed rank; reparameterizing the survivors through inverse Fourier or wavelet transforms yields SeLoRA, which beats LoRA, DoRA, and HiRA on reasoning and code with fewer parameters.","keywords":["parameter redundancy","low-rank adaptation","spectral encoding","sparsity property","Fourier transform","wavelet transform","parameter-efficient fine-tuning","large language models"],"falsifier":"Run the Figure 1 experiment as a sweep: fine-tune LLaMA-2-7B and LLaMA-3-8B at rank 32 with random masks of 20, 40, 60, and 80 percent across at least five seeds, and additionally redraw the shared index set $\\Omega$ several times at a fixed sparsity level. If dense-to-masked accuracy differences at 60 percent exceed seed-level noise, or if accuracy varies substantially across draws of $\\Omega$, the sparsity property and SeLoRA's motivation fail; stable parity would confirm the paper's central claim.","tokens_in":20921,"feed_emoji":"🌊","tokens_out":9070,"duration_ms":80720,"temperature":0.7,"pith_summary":"The paper claims that Low-Rank Adaptation (LoRA) carries heavy parameter redundancy, and that the redundancy can be exploited by sparsifying the parameters rather than shrinking the rank. At a fixed rank of 32, randomly zeroing out up to 60 percent of LoRA's trainable parameters preserves accuracy comparable to the dense adapter, while cutting rank alone degrades performance; the paper calls this the sparsity property. Building on it, the paper proposes SeLoRA, which reparameterizes the low-rank matrices as inverse spectral transforms (Fourier or wavelet) of a sparse set of learnable coefficients at a fixed random index set shared across modules. Across commonsense reasoning, math reasoning, and code generation on LLaMA-2-7B and LLaMA-3-8B, SeLoRA reports higher average accuracy than LoRA, DoRA, and HiRA while using roughly 40 percent fewer trainable parameters. The result matters because it suggests parameter efficiency and expressiveness in fine-tuning need not trade off, and that a spectral parameterization is a simple plug-in that improves existing LoRA variants.","feed_headline":"60% of LoRA weights can go silent — spectral encoding makes it win","feed_subtitle":"A new LoRA variant learns only sparse Fourier or wavelet coefficients and still beats LoRA, DoRA, and HiRA on reasoning and code.","key_machinery":"The load-bearing object is the reparameterization map $\\mathcal{T}$ applied to a sparsely masked spectral matrix: $\\tilde{A} = \\mathcal{T}(F_A)$ and $\\tilde{B} = \\mathcal{T}(F_B)$, where only the entries of $F_A, F_B$ at a fixed random index set $\\Omega$ are trainable and all others are frozen at zero, with $|\\Omega| = \\lfloor(1-\\eta)rd\\rfloor$ controlled by the sparse ratio $\\eta$. The map converts a sparse set of spectral coefficients into a dense spatial-domain matrix, so the adapter keeps LoRA's low-rank update structure while drawing its parameters from a spectral subspace. Initialization scales $F_A$ so that $\\text{Var}(\\mathcal{T}(F_A))$ matches a Xavier- or Kaiming-initialized auxiliary matrix, while $\\tilde{B}$ starts at zero, preserving LoRA's training stability. The machinery does two jobs at once: the masking experiments establish the sparsity property, and the spectral basis supplies the inductive bias that lets a sparse coefficient set reconstruct a high-quality dense adapter.","core_discovery":"The central claim is that a low-rank adapter can be learned inside a sparse spectral subspace without losing expressiveness: the update is written as $W' = W_0 + \\tilde{B}\\tilde{A}$ with $\\tilde{A} = \\mathcal{T}(F_A)$ and $\\tilde{B} = \\mathcal{T}(F_B)$, where $F_A$ and $F_B$ are spectral matrices whose learnable entries sit only at a randomly chosen index set $\\Omega$ shared across all modules, and $\\mathcal{T}$ is either the real part of the inverse 2D discrete Fourier transform or the inverse 2D wavelet transform (Haar by default). At rank 32, both SeLoRA variants match or exceed LoRA's accuracy on eight commonsense benchmarks while using about 0.28-0.5 percent of model parameters versus LoRA's 0.70-0.83 percent, and the wavelet variant raises average accuracy on math and code benchmarks by roughly 2 points. The paper further shows the plug-in property: wrapping DoRA and HiRA in the same spectral reparameterization (SeDoRA, SeHiRA) improves those baselines as well. A subspace analysis reports that SeLoRA's updates amplify task-relevant directions of $W$ less strongly than LoRA's, cutting the reverse amplification factor from 0.17 to 0.04, which the authors read as evidence that spectral encoding changes what LoRA learns, not merely how it is stored.","pith_inferences":["If the sparsity property generalizes, it suggests a spectral lottery-ticket picture: a fixed random mask in the spectral domain acts like a static sparse connectivity pattern, with the basis rather than the mask carrying the expressiveness; a testable extension is whether data-dependent masks chosen by gradient sensitivity beat the random $\\Omega$ at extreme sparsity.","The fourfold drop in reverse amplification factor (0.17 to 0.04) hints that spectral encoding may reduce interference with pretrained knowledge, which would predict that SeLoRA-style adapters forget less under sequential fine-tuning, a test the paper does not run.","Since SeLoRA with $\\eta=0$ differs from plain LoRA only in the basis, comparing them at matched parameter counts isolates what the spectral prior itself buys; the paper's gains at a fixed budget already suggest the prior, not just the sparsity, is doing the work.","The shared $\\Omega$ across modules means the model learns one common sparse spectral subspace for all adapted weights; a natural stress test is whether $\\Omega$ can be frozen at initialization and reused across tasks or backbones without retuning."],"forward_implications":["Under the sparsity property, density rather than rank is the right lever for shrinking LoRA: at rank 32, SeLoRA holds LoRA-grade accuracy with roughly 40 percent fewer trainable parameters (0.50 percent vs 0.83 percent on LLaMA-2-7B).","SeLoRA acts as a generic plug-in: the same spectral wrapper improves DoRA and HiRA as well, with SeDoRAW and SeHiRAW adding up to 1.8 and 2.1 points over their bases.","Spectral encoding buys the expressiveness of much larger ranks: SeLoRA at $r=32$ matches plain LoRA at $r=256$ on LLaMA-3-8B commonsense reasoning.","SeLoRA converts data into accuracy more efficiently: with 25 percent of the training data it already beats LoRA trained on the full set, and the gap widens as data grows.","Wavelet bases are the more robust instantiation: SeLoRAW outperforms SeLoRAF on most benchmarks, while the choice among Haar, Daubechies-4, Biorthogonal, and Coiflets changes results only marginally."],"supporting_citations":[{"why":"LoRA is the base method whose update rule $W' = W_0 + BA$ SeLoRA reparameterizes; its parameter counts and accuracy define the primary baseline.","marker":"Hu et al., 2021"},{"why":"DoRA is the integrable baseline that SeLoRA wraps into SeDoRA and must beat; its reproduced results set the comparison standard.","marker":"Liu et al., 2024b"},{"why":"HiRA is the high-rank Hadamard baseline that SeLoRA wraps into SeHiRA; the paper implements it from the original configuration since its code is not public.","marker":"Huang et al., 2025"},{"why":"FourierFT is the Fourier-parameterization baseline on math and code tasks that SeLoRAF is compared against under a similar parameter budget.","marker":"Gao et al., 2024"},{"why":"LS-LoRA supplies the sparse-reparameterization baseline whose masked higher-rank adaptation SeLoRA builds on and outperforms.","marker":"He et al., 2022"},{"why":"Cited for learning weight matrices from sparse spectral entries, providing the rationale for spectral reparameterization.","marker":"Wolter et al., 2020"},{"why":"Cited as prior work training networks in a compressed spectral weight space, supporting the expressiveness claim for spectral bases.","marker":"Irie and Schmidhuber, 2021"},{"why":"Supplies the wavelet-encoding construction that SeLoRAW instantiates with the inverse wavelet transform.","marker":"Van Steenkiste et al., 2016"}],"fun_headline_variants":["Spectral encoding slims LoRA, boosting math, code, and reasoning","SeLoRA: sparse spectral adapter beats LoRA with half the parameters","SeLoRA: spectral encoding makes LoRA leaner and pluggable","Sparse spectral basis boosts LoRA, DoRA, and HiRA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on the claim that randomly masking up to 60 percent of LoRA's parameters at rank 32 genuinely costs nothing in accuracy; if that parity is a fluke of one seed, one rank, or the particular sparsity levels chosen, then learning through a fixed random spectral mask has no reason to work, and the paper reports no sensitivity analysis for the random index set $\\Omega$ that every module shares.","fun_headline_variants_meta":{"raw":{"variants":["Spectral encoding slims LoRA, boosting math, code, and reasoning","SeLoRA: sparse spectral adapter beats LoRA with half the parameters","SeLoRA: spectral encoding makes LoRA leaner and pluggable","Sparse spectral basis boosts LoRA, DoRA, and HiRA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000522,"raw_usage":{"total_tokens":2570,"prompt_tokens":1035,"completion_tokens":1535,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":1454}},"tokens_in":651,"tokens_out":1535,"duration_ms":12875,"temperature":1.0,"reasoning_tokens":1454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:18:07.650781+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Figure 1 experiment as a sweep: fine-tune LLaMA-2-7B and LLaMA-3-8B at rank 32 with random masks of 20, 40, 60, and 80 percent across at least five seeds, and additionally redraw the shared index set $\\Omega$ several times at a fixed sparsity level. If dense-to-masked accuracy differences at 60 percent exceed seed-level noise, or if accuracy varies substantially across draws of $\\Omega$, the sparsity property and SeLoRA's motivation fail; stable parity would confirm the paper's central claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HiRA is the high-rank Hadamard baseline that SeLoRA wraps into SeHiRA; the paper implements it from the original configuration since its code is not public."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited for learning weight matrices from sparse spectral entries, providing the rationale for spectral reparameterization."},{"cited_title":"Training and Generating Neural Networks in Compressed Weight Space","cited_arxiv_id":"2112.15545","evidence_quote":"Cited as prior work training networks in a compressed spectral weight space, supporting the expressiveness claim for spectral bases."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the wavelet-encoding construction that SeLoRAW instantiates with the inverse wavelet transform."}],"review_version":2}