REVIEW 5 major objections 5 minor 2 cited by
LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning
T0 review · 5 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read This paper proves that a sparse, deliberately chosen set of frequency components can reconstruct fine-tuning weight updates more accurately than a low-rank decomposition of the same size.
desk verdict Worth engaging for the method and experiments, but the central theory rests on an i.i.d. Gaussian assumption that is assumed more than shown. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the sparse inverse discrete cosine transform reparameterization $\Delta W = \alpha C^T S(a,l,1) D$, where $C$ and $D$ are DCT matrices, $S$ scatters $B$ learnable coefficients $a$ onto locations $l$ of the DCT spectrum, and $\alpha$ scales the update. The theoretical machinery behind it is the Gaussian update model: Proposition 1 models the vectorized update as asymptotically i.i.d. normal, which turns reconstruction quality into a pure random-matrix calculation. Theorem 1 then compares expected residual energies: the best low-rank approximation keeps the $r$ largest singular values, while frequency methods keep the largest Fourier or DCT coefficients, and the paper shows that for i.i.d. Gaussian data the latter carries more expected energy per parameter. Theorem 2 identifies iDCT as an exact real-valued implementation of the individually optimal Fourier coefficient selection. To make locations learnable, the paper uses a central-difference surrogate gradient that measures how much the reconstruction changes when a coefficient is shifted by one bin, and an alternating coefficients-then-locations optimization finds the informative part of the spectrum early in training.
What would settle it
Take a real update matrix $\Delta W$ from a fine-tuned model, such as a query or value projection in LLaMA fine-tuned on Alpaca, and compare the paper's reconstruction ordering directly: for each layer, compute the expected squared Frobenius error of the best rank-$r$ approximation and of keeping the $r$-largest DCT coefficients under the same parameter accounting. If for $r<K/3$ the low-rank error is smaller on the majority of layers, or if the eigenvalue mass of the sample correlation matrix outside the Marchenko-Pastur bounds exceeds the paper's critical values, the Gaussian-model assumption fails in a way that invalidates the ordering.
Extended reading notes
Core claim
The paper's central claim is that, with an equal number of parameters, a sparse representation of a weight update in a frequency basis can have lower expected reconstruction error than the best low-rank representation, provided the frequency components are selected by magnitude rather than at random. Concretely, Theorem 1 states that for a $K\times K$ weight matrix with i.i.d. standard normal entries and rank $r<K/3$, the expected squared Frobenius error of random Fourier coefficient selection exceeds that of low-rank approximation, which in turn exceeds that of selecting the largest-amplitude Fourier locations, which in turn exceeds that of selecting the largest individual real and imaginary Fourier coefficients. Theorem 2 states that selecting the largest coefficients of the discrete cosine transform attains exactly the same expected error as that individually optimal Fourier selection, while working entirely with real numbers. The proposed LoCA method implements this insight by reparameterizing $\Delta W$ through iDCT, learning both the coefficients and the discrete locations, with location gradients estimated by central finite differences and an alternating optimization schedule. The method, tested on GLUE, E2E, instruction tuning, and image classification, matches or beats low-rank baselines while using far fewer parameters, and consistently beats FourierFT at equal budget.
Load-bearing premise
The ordering rests on treating a fine-tuning weight update as a matrix of independent, identically distributed Gaussian entries; if real updates carry low-rank or correlated structure, the frequency advantage could shrink or reverse.
Editorial extensions
If this is right
- For the same parameter count, a sparse frequency-domain update can represent a fully fine-tuned $\Delta W$ more faithfully than a low-rank update, so PEFT budgets can be smaller without sacrificing expressivity.
- Under the paper's Gaussian model, random frequency-component placement is provably weaker than low-rank at the same budget; selection must be deliberate.
- Because DCT selection matches optimal Fourier coefficient selection, real-valued iDCT is the practical way to realize the expressivity advantage without complex-valued optimization.
- Dynamic location search during training offers a way to allocate the parameter budget to the spectrum's most informative bins, adapting per weight matrix.
- On language and vision benchmarks, retaining the same component budget yields performance at or above LoRA-family baselines with a fraction of the parameters.
Reading between the lines
- Editorial inference: the theorem's ordering depends on updates being diffuse Gaussian noise; on tasks where $\Delta W$ is genuinely low-rank or strongly correlated, LoRA may retain its edge, and the method's advantage is most likely for high-dimensional, noisy updates.
- Editorial inference: the finite-difference location gradient is a general mechanism that amounts to sparse support search in any orthogonal basis, so applying the same selection scheme to wavelets or learned dictionaries would test whether DCT is special or merely convenient.
- Editorial inference: the paper's appendix correlation analysis suggests the ordering tolerates mild uniform correlations among update entries; a stronger test would measure the actual covariance structure of $\Delta W$ across layers and check whether its eigenvalue mass outside the Marchenko-Pastur interval exceeds the identified critical levels.
- Editorial inference: if frequency selection is superior for i.i.d. updates, parameter budgets could be allocated adaptively per layer by the location-gradient signal, concentrating components where the spectrum is richest.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces LoCA, a parameter-efficient fine-tuning method that reparameterizes weight updates as a sparse inverse-DCT with learnable coefficient values and locations. The authors first claim, via Proposition 1, that weight updates during fine-tuning are asymptotically i.i.d. Gaussian, and then use this to prove (Theorem 1) that, under an i.i.d. Gaussian model, sparse frequency-domain reconstruction with carefully selected components has lower expected Frobenius error than low-rank reconstruction at comparable parameter budgets, and that random frequency selection is worse than low-rank. Theorem 2 shows the DCT-based selection is equivalent to the optimal Fourier coefficient selection. The method alternates between optimizing coefficients and locations, using central-difference gradient estimates for the discrete locations. Experiments on GLUE, E2E, Alpaca instruction tuning, image classification, and Stable Diffusion are reported.
Significance. If the theoretical results were fully established, the paper would provide the first formal expressivity comparison between frequency-domain and low-rank PEFT, and LoCA itself shows consistently competitive or superior performance to FourierFT and near-LoRA results with fewer parameters. The connection to random matrix theory (Wishart eigenvalues, chi-square order statistics) is a useful framework. However, the central theorem is not fully proven and relies on assumptions that are not adequately validated, and the parameter accounting in both the theory and experiments needs correction. These issues currently limit the strength of the claims.
major comments (5)
- [Appendix G, Theorem 1, Eq. (15)] The key inequality (15) that establishes the ordering between low-rank and frequency methods is only verified numerically for general r and K; the analytic argument covers only r=1 and large K. Since Theorem 1 is stated as a theorem for all r < K/3, numerical verification is insufficient. This is load-bearing because the full ordering in Theorem 1 depends on (15). The authors should either provide a complete proof of (15) or explicitly restate the result as a conjecture supported by numerical evidence, and adjust the abstract and discussion accordingly.
- [Section 2, Proposition 1, Appendix A/B] Assumptions A1 and A2 in Proposition 1 are load-bearing but not adequately established. The hypothesis test in Appendix B only checks closeness of the empirical distribution to a Gaussian under a synthetic null of perturbed Gaussians; it does not test independence. The Marchenko-Pastur fit is consistent with i.i.d. entries but also with many structured ensembles such as low-rank spikes or block-correlated matrices. The paper itself concedes in Appendix A that strict i.i.d. of trained neural-network parameters is an open question. To support Theorem 1, the authors should directly measure structure in real fine-tuning updates (e.g., singular value decay, effective rank, correlation between entries) or empirically verify the Theorem 1 ordering on actual full fine-tuning updates.
- [Section 3, Theorem 1, parameter budget for W_F^(2)] The comparison for W_F^(2) is not at the same parameter budget. In the theorem, N0 = (p+q)r = 2Kr for a square matrix, and N2 = 2/3 N0 locations. Each selected location in the non-redundant half carries two real coefficients (real and imaginary parts), so W_F^(2) uses 2N2 = 4/3 N0 learnable parameters, whereas W_R uses N0 parameters. Thus the inequality E[L(W, W_R)] > E[L(W, W_F^(2))] is an unequal-budget comparison and does not establish the claimed expressivity superiority. To make the comparison fair, N2 should be set to 1/2 N0 so that the total coefficient count equals N0, and the proof and numerical verification should be redone under that budget.
- [Section 5, Implementation Details, Tables 1-4] The reported parameter counts for LoCA exclude the location variables l, which are optimized for Bs steps and must be known to the model to place the learned coefficients. This undercounts trainable parameters and makes the comparisons with FourierFT (which has fixed random locations) and with LoRA potentially misleading. The authors should either include l in the reported parameter counts, or provide a concrete argument that the information content of the learned locations is negligible in the stated budgets and is not providing an unfair advantage over the baselines.
- [Appendix P, non-i.i.d. analysis] The robustness analysis only considers the uniform correlation model Sigma = rho * 1*1^T + I. Real fine-tuning updates are more plausibly low-rank or block-structured, which is exactly the regime most relevant to a comparison against low-rank adaptation. The authors should either study structured dependence models (e.g., a rank-k spike plus noise, or block correlations) or empirically test whether the Theorem 1 ordering holds on real weight updates. As presented, the reported critical rho_c values do not address the concern that structured dependencies may invalidate the i.i.d. premise of the theory.
minor comments (5)
- [Table 4] In the ViT-large LoCA 144K row, the RESISC45 entry '94.41±91.76' is clearly a typo and should be corrected.
- [Appendix M] The text contains 'mean squeue error', which should be 'mean squared error'.
- [Figure 3 caption] The caption mentions 'FHVC', which should be 'FGVC'.
- [Section 2] The empirical observations supporting the Gaussian model are based on a single model (LLaMA-7b) fine-tuned on a single dataset (Alpaca-52K); the authors should acknowledge this limited basis or include additional models/tasks in the analysis.
- [Appendix B] The description of the hypothesis test is somewhat informal; in particular, the construction of the null distribution by sampling from perturbed Gaussians should be more clearly distinguished from a proper bootstrap, and the choice of epsilon and the perturbation variance should be justified.
Circularity Check
No significant circularity: the paper's theoretical claims are derived under an explicit Gaussian model, and the acknowledged i.i.d. assumption is a stated premise rather than a fitted prediction.
full rationale
The derivation chain is: empirical observation of near-Gaussian ΔW → Proposition 1 (conditional on Assumptions A1 and A2) → Theorem 1 (reconstruction-error ordering under W ~ N(0, I)) → Theorem 2 (DCT/Fourier equivalence) → the LoCA method. Proposition 1 is explicitly conditional: it says 'Under these assumptions' and obtains the distribution of ΔW by adding a Gaussian prior (A1) to an asymptotically normal M-estimator residual (A2). This is a standard statistical model, not a hidden reuse of the conclusion. The paper is transparent that A2's i.i.d. condition is the strongest assumption: Appendix A states that 'the strict i.i.d. property of parameters in trained neural networks remains a subject of ongoing research,' and the MP-law and hypothesis-test evidence is presented as motivation, not as a circular derivation. Theorem 1 then performs non-trivial probabilistic work — Wishart densities, order statistics of χ² variables, Tracy-Widom asymptotics, and numerical verification for finite r and K — so its inequality does not reduce to the Gaussian assumption by construction. Theorem 2 is an independent orthogonality/Parseval argument for the DCT. There are no load-bearing self-citations: the cited M-estimator and random-matrix results are external. The main weakness, that real fine-tuning updates may be low-rank or correlated rather than i.i.d. Gaussian, is a correctness/external-validity concern about whether the model transfers, not a circularity in the derivation itself. The partial reliance on numerical integration in the proof of Theorem 1 is a rigor gap, not a circular step. Overall, the paper's central claims are conditional on a clearly stated, albeit imperfectly validated, statistical model, and the ranking results contain independent mathematical content.
Assumptions & free parameters
free parameters (3)
- Number of frequency components B =
1000 to 150,000 depending on task and model
- Scaling value alpha =
0.5, 1, 16, 64 depending on task
- Alternating optimization steps Bs =
10% to 20% of total training steps (e.g., 600 to 3000)
assumptions (3)
- ad hoc to paper A1: For real-world fine-tuning datasets, the vectorized parameter W^V follows a Gaussian prior centered at the pre-trained W0^V.
- ad hoc to paper A2: W' is an M-estimator of W with asymptotically normal estimation error, the entries of W'-W are asymptotically i.i.d., and W'-W is independent of W.
- ad hoc to paper The inequality in Eq. (15) holds for all r < K/3.
Cite this review
Pith. "Pith review of LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning." pith.science (2026). https://pith.science/paper/HYHMAPFK
@misc{pith2026250206820,
author = {Pith},
title = {Pith review of: LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning},
year = {2026},
howpublished = {\url{https://pith.science/paper/HYHMAPFK}},
note = {Machine review of arXiv:2502.06820}
}
read the original abstract
Low-rank adaptation (LoRA) has become a prevalent method for adapting pre-trained large language models to downstream tasks. However, the simple low-rank decomposition form may constrain the hypothesis space. To address this limitation, we introduce Location-aware Cosine Adaptation (LoCA), a novel frequency-domain parameter-efficient fine-tuning method based on inverse Discrete Cosine Transform (iDCT) with selective locations of learnable components. We begin with a comprehensive theoretical comparison between frequency-domain and low-rank decompositions for fine-tuning pre-trained large models. Our analysis reveals that frequency-domain decomposition with carefully selected frequency components can surpass the expressivity of traditional low-rank-based methods. Furthermore, we demonstrate that iDCT offers a more efficient implementation compared to inverse Discrete Fourier Transform (iDFT), allowing for better selection and tuning of frequency components while maintaining equivalent expressivity to the optimal iDFT-based adaptation. By employing finite-difference approximation to estimate gradients for discrete locations of learnable coefficients on the DCT spectrum, LoCA dynamically selects the most informative frequency components during training. Experiments on diverse language and vision fine-tuning tasks demonstrate that LoCA offers enhanced parameter efficiency while maintains computational feasibility comparable to low-rank-based methods.
Figures
Figures from the paper (10 more)
Forward citations
Cited by 2 Pith papers
-
PFMBench: Protein Foundation Model Benchmark
A comprehensive benchmark of 17 protein foundation models across 38 tasks yields task correlations, a streamlined protocol, and identifies ProTrek as the strongest general performer.
-
NAN: A Training-Free Solution to Coefficient Estimation in Model Merging
NAN sets merging coefficients inversely proportional to each model's parameter norm and claims a least-squares justification, but the derivation yields a different formula and performance gains are inconsistent.
Reference graph
Works this paper leans on
-
[1]
From the observed ∆W , compute the empirical mean ˆµ and variance ˆσ2
-
[2]
Generate 1e5 samples from N (w; ˆµ, ˆσ2), denoted this set of samples by G
-
[3]
• Calculate the empirical distribution of the perturbed samples
Generate B perturbed distributions: • Add small random perturbations e ∼ N(e; 0, σe2) to the M samples, where σe = 1e−5. • Calculate the empirical distribution of the perturbed samples. • Compute the total variation distance between the obtained empirical distribution and G. • If the total variation distance is less than ϵ, keep this distribution. • Repea...
-
[4]
For each of the 100 perturbed distributions: • Sample 10 sets of p × q points. • For each set, calculate the total variation distance between the empirical distribution of this set and G. This results in M × P total variation distances, forming the distribution of the test statistic under H0
-
[5]
Calculate the total variation distance between the empirical distribution of ∆W and G, denoted by T
-
[6]
The p-value is the percentile of T in the M × P total variation distances
-
[7]
Reject H0 if the p-value is less than the chosen significance level (e.g., 0.05). Otherwise, accept H0. Note that although this process is not strictly a bootstrap (as it does not directly resample from the original data), it does use the idea of repeated sampling to generate the distribution of the test statistic. Traditional bootstrap typically resample...
work page 2024
-
[8]
For r >1 we can apply similar analysis but that will be much more complex
For r = 1 but not sufficiently large K, we directly calculate the Eλ′ 1 and compare it with 2K. For r >1 we can apply similar analysis but that will be much more complex. We demonstrate the result in later numerical approximation (Fig. 6 and 7). Now we turn to L(i) F , i= 1, 2, 3. Remember we have F (W ) = {Re(H) + iIm(H)} W {Re(H) + iIm(H)} = {Re(H)W Re(...
work page 2025
Show all 13 references
-
[9]
Define ψ1,
Thus we can reformulate L(2) F and L(3) F in a more clear way. Define ψ1, . . . , ψK2 i.i.d. ∼ χ2 1, ϕ1, . . . , ϕ(K2−4)/2 i.i.d. ∼ χ2 2, ϕ(K2−2)/2, . . . , ϕ(K2+4)/2 i.i.d. ∼ χ2
-
[10]
Change the natural image to a cartoon- style image
Denote the order statistics of ψi, ϕi as ψ(1) ≥ . . .≥ ψ(K2) and ϕ(1) ≥ . . .≥ ϕ((K2+4)/2), we then have L(2) F d. = (K2+4)/2X i=N2+1 ϕ(i), L(3) F d. = K2 X i=N3+1 ψ(i) , (14) where d. = means equality in distribution. In other words, EW ∼G K 2 − L(2) F = N2X i=1 Eϕ(i), EW ∼G ...
1979
-
[11]
The critical correlation values where method performance characteristics change are statis- tically significant and detectable using standard random matrix theory diagnostics
-
[12]
The monotonic increase in critical correlation with rank suggests that higher-dimensional representations are more robust to dependencies
-
[13]
These results demonstrate that while strict independence is violated in practice, our theoretical in- sights remain valid under realistic levels of parameter dependency
Even under substantial and detectable correlations, the performance advantages of frequency-domain methods persist, supporting the practical validity of our theoretical framework. These results demonstrate that while strict independence is violated in practice, our theoretical...
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.