REVIEW 4 major objections 4 minor 15 references
MoKA: Mixture of Kronecker Adapters
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read MoKA models a PEFT weight update as a gated mixture of Kronecker products and claims this restores quantized LLaMA accuracy with up to 27x fewer trainable parameters than QLoRA.
desk verdict Gated mixture of Kronecker adapters is a plausible PEFT idea with decent-looking gains, but the advertised convergence theorem does not follow from the appendix proof, and the gating claim is overstated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the mixture-of-Kronecker residual $\Delta W = \sum_{i=1}^r \alpha_i (A_i \otimes B_i)$, with softmax-normalized gating scalars $\alpha_i$ and factor matrices $A_i$, $B_i$ of different shapes. The argument relies on the identity $(A\otimes B)x = V(B R_{n_b,n_a}(x) A^\top)$, which replaces the explicit Kronecker product with a reshape followed by two matrix multiplications, letting the method ride on GPU-optimized dense kernels. Padding and truncation let each factor pair work even when the product of its input dimensions does not exactly match the layer width. For the MoKAs variant, setting $A_i = I$ reduces the mixture to block-diagonal updates, exploiting the observation that nearby tokens dominate attention. The accompanying convergence claim is carried by Theorem 1, which decomposes the loss into optimization and approximation errors and bounds the average squared gradient norm by $(L(u_0)-L_{\min})/(\eta T) + \eta L G / 2$.
What would settle it
Run SGD on a 4x4 linear regression problem with $U$ constrained to a single Kronecker product $A\otimes B$ and compute $\mathbb{E}[\langle \nabla L(u_t), G_t\rangle] - \mathbb{E}[\|\nabla L(u_t)\|^2]$ at each step; a nonzero difference while assumptions A1-A3 hold would show the theorem's replacement step is false. Separately, a seed-variance replication of the commonsense reasoning table on one dataset would tell whether the roughly one-percent margin over QLoRA is reproducible.
Extended reading notes
Core claim
On the authors' own terms, the central discovery is that a gated mixture of Kronecker products, $\Delta W = \sum_i \alpha_i (A_i \otimes B_i)$, can serve as the task-specific residual in a frozen large language model and achieves better accuracy than QLoRA and QDoRA with far fewer trainable parameters. The mixture weights $\alpha_i$ come from a softmax over learnable gating scalars, and each Kronecker factor pair can have a different shape, so the update is not confined to a fixed rank or block structure. A special case called MoKAs fixes $A_i$ to the identity, turning the update into a mixture of block-diagonal matrices that exploits the local-token bias of attention. The authors show that the Kronecker computation can be rewritten through reshape and matrix multiplication, making it GPU-friendly, and they report experiments on 4-bit quantized LLaMA models. They also give a proof that SGD with such Kronecker-structured updates converges at $O(1/\sqrt{T})$.
Load-bearing premise
The load-bearing premise is the proof step that treats the Kronecker-structured gradient $G_t$ as an approximation of the true gradient and replaces the inner product $\langle \nabla L(u_t), G_t\rangle$ with $\|\nabla L(u_t)\|^2$; if that identification does not hold, the claimed $O(1/\sqrt{T})$ convergence rate is not established.
Editorial extensions
If this is right
- MoKA and MoKAs give the practitioner a parameter-efficiency and accuracy frontier that improves on QLoRA and QDoRA in the reported instruction-tuning and commonsense settings, so users of quantized LLMs can spend fewer trainable parameters for the same or better accuracy.
- The effective update is no longer capped by a manually chosen rank, because the rank of a Kronecker product is the product of the factor ranks, letting the adapter family represent full-rank updates while remaining small.
- Because the Kronecker operation is re-expressed as standard matrix multiplications, the method inherits the speed and memory behavior of dense GEMM kernels rather than requiring specialized Kronecker hardware.
- If Theorem 1 is accepted, Kronecker-structured fine-tuning has the same $O(1/\sqrt{T})$ SGD convergence rate as unconstrained fine-tuning, suggesting the structure does not slow optimization in the worst case.
- The MoKAs variant shows that a mixture of identity-Kronecker (block-diagonal) updates is enough to match the full mixture on several tasks, pointing to local-token structure as a key ingredient.
Reading between the lines
- The gating scalars $\alpha_i$ are global learnable constants per layer, not functions of the input $x$; the weighting is dynamic across layers and training steps but static across tokens within a layer, so an input-conditioned gating network is a natural next step that the paper does not test.
- The proof of Theorem 1 hinges on an unproven identification: the sum of Kronecker products of factor gradients, $G_t$, is treated as the true gradient $\nabla L(u_t)$, and the inner product $\langle \nabla L, G_t\rangle$ is replaced by $\|\nabla L\|^2$; this equality does not hold in general, so the theoretical convergence claim should be read as conditional on an approximation whose error is not
- Because padding and truncation discard positions at the end of the sequence representation, the method may interact differently with architectures where the last token carries classification or generation information; a per-task study of where truncation falls would test whether this matters.
- The same gated-mixture idea could generalize to other structure families, replacing a single adapter type with a small palette of structured updates (low-rank, block, circulant) and letting the optimizer pick the right structure per layer.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MoKA, a parameter-efficient fine-tuning method that models weight updates as a gated mixture of Kronecker products, together with a lightweight variant MoKAs that uses block-diagonal structure. The method is evaluated on instruction-tuning and commonsense reasoning benchmarks with 4-bit quantized LLaMA2-7B and LLaMA3-8B, showing accuracy gains over QLoRA and QDoRA with substantially fewer trainable parameters. The paper also presents a convergence theorem (Theorem 1) claiming that SGD with Kronecker-structured updates achieves the same O(1/sqrt(T)) rate as unconstrained fine-tuning.
Significance. If the empirical findings are reproducible, MoKA offers a practically useful parameter-accuracy trade-off, and the reformulation of Kronecker computations as standard matrix multiplications is a valuable engineering contribution. The gating ablation in Table 3 provides some evidence that the mixture weighting helps. However, the central theoretical claim is not established: the proof of Theorem 1 contains an unjustified and generally false identification of the structured update with the full gradient. Since the convergence guarantee is advertised as a key contribution, the paper's theoretical contribution is currently unsupported. The empirical comparison is also limited to two baselines, omitting the closest Kronecker-based competitors.
major comments (4)
- [Appendix, Proof of Theorem 1] The step that replaces E[<∇L(u_t;ξ_t), G_t>] with E[||∇L(u_t;ξ_t)||^2_F] is unjustified and generally false. For the parameterization U = Σ_i A_i ⊗ B_i, the leading-order structured update is G_t = J J^T ∇L (the projection of ∇L onto the tangent space of the Kronecker-structured manifold), so <∇L, G_t> equals the squared norm of that projection, not ||∇L||^2. A concrete counterexample: take L(u) = ½||u - I||_F^2 on 2×2 matrices with r=1, A ∈ R^{2×1}, B ∈ R^{1×2} initialized at zero. Then ∇L(0) = -I, but G_0 = 0 because both factors are zero, so <∇L(0), G_0> = 0 while ||∇L(0)||^2 = 2. The descent inequality used in the proof would predict a decrease in loss, but the actual factor-space update leaves the loss unchanged. Therefore the telescoping sum and the claimed O(1/√T) rate do not follow from assumptions A1–A3.
- [Appendix, Proof of Theorem 1] The proof analyzes a linearized update u_{t+1} − u_t ≈ −ηG_t, but the actual algorithm updates the Kronecker factors as A_i ← A_i − η∇_{A_i}L and B_i ← B_i − η∇_{B_i}L, which produces an additional second-order term η^2 ∇_{A_i}L ⊗ ∇_{B_i}L in the change of U. The descent inequality is applied to the approximate update, not to the actual update performed by MoKA. Consequently, even the theoretical object being analyzed is not the algorithm implemented in the experiments, and Theorem 1 does not establish convergence of MoKA as described in Section 3.
- [Introduction and Equation (6)] The gating mechanism is described as 'dynamically weighs the contribution of each adapter based on input context' and 'adaptively selects and weighs Kronecker factors,' but Equation (6) defines α_i = softmax(g)_i with scalar g_i that are independent of the input. After training, these weights are fixed constants, not input-dependent. This overstates the adaptivity of the method and should be corrected to describe the gating as a set of learnable scalar mixture weights rather than a dynamic input-conditioned mechanism.
- [Experiments and Results] The empirical evaluation compares MoKA only against QLoRA and QDoRA. Since MoKA is a Kronecker-based adapter, the closest baselines are KronA and AdaKron, which are discussed in Related Work but never included in the experiments. Without comparisons to these methods, the claim of a state-of-the-art trade-off among Kronecker adapters is not fully supported. Adding these baselines would strengthen the empirical contribution.
minor comments (4)
- [Assumptions in Theorem 1] Assumption A1 (convexity) is never used in the proof; the argument relies only on L-smoothness and the boundedness assumption A3. The authors should either use convexity or remove it from the assumptions.
- [Section 2.2 Theoretical analysis] The decomposition L(u) − L* = (L(u) − L_min) + (L_min − L*) is clear, but the theorem bounds the average squared gradient norm in terms of L_min, not L*. The relationship between the bound and the decomposition should be stated explicitly to avoid confusion.
- [MoKAs experimental settings] The choice of prime p for the MoKAs variant is unexplained; the text says 'p is a prime number in the range 2–97' but gives no reason for the primality constraint or the specific range. A brief justification would improve clarity.
- [Table 3] The improvements from gating are small (roughly 0.2 to 0.9 points) and no variance or significance measures are reported. A statement about the stability of these gains across seeds would make the gating ablation more convincing.
Circularity Check
No circular derivation: MoKA's adapters are learned and compared externally; the Theorem 1 proof issue is a correctness gap, not circularity.
full rationale
MoKA's central claims are not circular. The proposed update Delta W = sum_i alpha_i (A_i tensor B_i) has all parameters (gating weights alpha_i and Kronecker factors A_i, B_i) learned from task data, and the empirical results are measured against external baselines (QLoRA, QDoRA) on standard benchmarks, so the reported improvements are not fitted inputs renamed as predictions. The parameter-count reduction (up to 27x) follows directly from the chosen factor dimensions rather than from an outcome-dependent fit. The theoretical section is an a priori optimization bound, not a derivation from the experimental results. The proof of Theorem 1 does contain a serious unsupported step: in the Appendix, paragraph beginning 'Considering that G_t is an approximation', the text replaces E[<grad L, G_t>] with E[||grad L||^2] without justification. This substitution is not implied by assumptions A1-A3 and is generally false; it invalidates the advertised O(1/sqrt(T)) rate. That is a mathematical correctness flaw in a load-bearing theoretical contribution, but it is not circularity: the claim is not equivalent to its inputs by construction, and the proof does not presuppose the theorem's conclusion through a definition or fitted parameter. The only self-citations (KronA and related Kronecker-adapter work by the same group) are used as background for the Kronecker parametrization and the rank(A tensor B) = rank(A) * rank(B) fact, which is independently established; no load-bearing uniqueness or convergence theorem is imported from the authors' prior work. Therefore the paper has no significant circularity, only a non-circular technical flaw and a minor non-load-bearing self-citation.
Assumptions & free parameters
free parameters (4)
- Kronecker filter shapes (LLaMA2-7B) =
(64x64,64x64), (32x128,128x32), (128x32,32x128), (16x256,256x16), (256x16,16x256), each instantiated twice
- Kronecker filter shapes (LLaMA3-8B value projection) =
(32x64,32x64), (16x128,64x32), (64x32,16x128), (8x256,128x16), (128x16,8x256), each twice
- Number of mixture components r =
10 per projection (5 shapes times 2)
- MoKAs prime p range =
primes in 2..97
assumptions (3)
- standard math Kronecker product identity (A⊗B)x = V{B R_{n_b,n_a}(x) A^T} (Van Loan and Pitsianis)
- domain assumption Loss L is convex and L-smooth (A1), stochastic gradients are unbiased (A2), Kronecker factors and their gradients are bounded (A3)
- ad hoc to paper The factor-space update direction G_t can be identified with the unconstrained gradient ∇L(u_t), so that E[⟨∇L,G_t⟩] = E[||∇L||^2]
Cite this review
Pith. "Pith review of MoKA: Mixture of Kronecker Adapters." pith.science (2026). https://pith.science/paper/OODNE7MT
@misc{pith2026250803527,
author = {Pith},
title = {Pith review of: MoKA: Mixture of Kronecker Adapters},
year = {2026},
howpublished = {\url{https://pith.science/paper/OODNE7MT}},
note = {Machine review of arXiv:2508.03527}
}
read the original abstract
Parameter-efficient fine-tuning (PEFT) is essential for reducing the computational overhead of large language models (LLMs). Low-rank family adapters are commonly used to control the parameter size efficiently while maintaining the generative power of LLMs. However, their limited expressiveness due to the rank constraint often restricts their performance on complex tasks. We propose Mixture of Kronecker Adapters (MoKA), a new generation of Kronecker adapters that addresses this limitation by modeling weight updates as a mixture of Kronecker products. Our proposed adapter leverages a gating mechanism that measures the importance of each Kronecker factor, enabling more expressive adaptation. Moreover, MoKA enables a rank flexibility that provides a better trade-off between parameter efficiency and accuracy. To ensure hardware efficiency, we reformulate Kronecker computations using standard matrix operations, allowing seamless deployment on GPU-optimized hardware. We conduct extensive experiments on instruction-tuning and commonsense reasoning tasks using low-bit quantized versions of LLaMA2-7B and LLaMA3-8B models. MoKA not only outperforms PEFT baselines, but also reduces the number of trainable parameters up to 27x, achieving state-of-the-art trade-offs between performance and parameter efficiency.
Figures
Reference graph
Works this paper leans on
-
[3]
KronA: Parameter Efficient Tuning with Kronecker Adapter. arXiv:2212.10650. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J
-
[5]
Unnatural instructions: Tuning language models with (al- most) no human labor.arXiv preprint arXiv:2212.09689. Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S
-
[7]
K¨opf, A.; Kilcher, Y .; V on R ¨utte, D.; Anagnostidis, S.; Tam, Z
Longform: Optimizing instruction tuning for long text generation with corpus extraction.arXiv preprint arXiv:2304.08460. K¨opf, A.; Kilcher, Y .; V on R ¨utte, D.; Anagnostidis, S.; Tam, Z. R.; Stevens, K.; Barhoum, A.; Nguyen, D.; Stan- ley, O.; Nagyfi, R.; et al
-
[9]
Liu, S.-Y .; Wang, C.-Y .; Yin, H.; Molchanov, P.; Wang, Y .- C
Prefix-tuning: Optimiz- ing continuous prompts for generation.arXiv preprint arXiv:2101.00190. Liu, S.-Y .; Wang, C.-Y .; Yin, H.; Molchanov, P.; Wang, Y .- C. F.; Cheng, K.-T.; and Chen, M.-H
-
[11]
Investigating Pub- lic Fine-Tuning Datasets: A Complex Review of Current Practices from a Construction Perspective.arXiv preprint arXiv:2407.08475. Mao, Y .; Mathias, L.; Hou, R.; Almahairi, A.; Ma, H.; Han, J.; Yih, W.-t.; and Khabsa, M
-
[12]
Rajabzadeh, H.; Valipour, M.; Zhu, T.; Tahaei, M
On the importance of local in- formation in transformer based models.arXiv preprint arXiv:2008.05828. Rajabzadeh, H.; Valipour, M.; Zhu, T.; Tahaei, M. S.; Kwon, H. J.; Ghodsi, A.; Chen, B.; and Rezagholizadeh, M
arXiv 2008
-
[13]
QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning. In Dernoncourt, F.; Preot ¸iuc-Pietro, D.; and Shimorina, A., eds.,Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing: Industry Track, 712–718. Miami, Florida, US: Association for Computational Linguistics. Shi, H.; Gao, J.; Ren,...
work page 2024
-
[14]
Dylora: Parameter efficient tuning of pre- trained models using dynamic search-free low-rank adapta- tion.arXiv preprint arXiv:2210.07558. Van Loan, C. F.; and Pitsianis, N
Show all 15 references
-
[15]
Zhang, Q.; Chen, M.; Bukharin, A.; He, P.; Cheng, Y .; Chen, W.; and Zhao, T
Self-instruct: Align- ing language models with self-generated instructions.arXiv preprint arXiv:2212.10560. Zhang, Q.; Chen, M.; Bukharin, A.; He, P.; Cheng, Y .; Chen, W.; and Zhao, T
-
[2019]
arXiv:1711.05101
Decoupled Weight De- cay Regularization. arXiv:1711.05101. Ma, R.; Li, W.; and Shang, F
-
[2020]
Honovich, O.; Scialom, T.; Levy, O.; and Schick, T
Measuring mas- sive multitask language understanding.arXiv preprint arXiv:2009.03300. Honovich, O.; Scialom, T.; Levy, O.; and Schick, T
2009 arXiv
-
[2021]
In Moens, M.-F.; Huang, X.; Specia, L.; and Yih, S
The Power of Scale for Parameter-Efficient Prompt Tuning. In Moens, M.-F.; Huang, X.; Specia, L.; and Yih, S. W.-t., eds.,Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 3045–3059. Online and Punta Cana, Dominican Republic: Association...
2021
-
[2022]
Ben Zaken, E.; Goldberg, Y .; and Ravfogel, S
Training a helpful and harmless assistant with re- inforcement learning from human feedback.arXiv preprint arXiv:2204.05862. Ben Zaken, E.; Goldberg, Y .; and Ravfogel, S
-
[2023]
In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 5254–5276
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Lan- guage Models. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 5254–5276. Singapore: Association for Computati...
2023
-
[2024]
In2024 Joint International Conference on Computational Linguistics, Language Resources and Evalu- ation, LREC-COLING 2024-Main Conference Proceedings, 350–357
Adakron: An adapter-based parameter efficient model tuning with kro- necker product. In2024 Joint International Conference on Computational Linguistics, Language Resources and Evalu- ation, LREC-COLING 2024-Main Conference Proceedings, 350–357. Dettmers, T.; Pagnoni, A.; Holtz...
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.