Pith. sign in

REVIEW 12 cited by

KronA: Parameter Efficient Tuning with Kronecker Adapter

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2212.10650 v1 pith:F2XV7NX4 submitted 2022-12-20 cs.CL

Ali Edalati , Marzieh Tahaei , Ivan Kobyzev , Vahid Partovi Nia , James J. Clark , Mehdi Rezagholizadeh This is my paper
classification cs.CL
keywords fine-tuningmethodsefficientkroneckerlow-rankmodelplmsadapter
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Fine-tuning a Pre-trained Language Model (PLM) on a specific downstream task has been a well-known paradigm in Natural Language Processing. However, with the ever-growing size of PLMs, training the entire model on several downstream tasks becomes very expensive and resource-hungry. Recently, different Parameter Efficient Tuning (PET) techniques are proposed to improve the efficiency of fine-tuning PLMs. One popular category of PET methods is the low-rank adaptation methods which insert learnable truncated SVD modules into the original model either sequentially or in parallel. However, low-rank decomposition suffers from limited representation power. In this work, we address this problem using the Kronecker product instead of the low-rank representation. We introduce KronA, a Kronecker product-based adapter module for efficient fine-tuning of Transformer-based PLMs. We apply the proposed methods for fine-tuning T5 on the GLUE benchmark to show that incorporating the Kronecker-based modules can outperform state-of-the-art PET methods.

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MoKA: Mixture of Kronecker Adapters

    cs.LG 2025-08 conditional novelty 6.0 of 10

    MoKA models weight updates as a gated mixture of Kronecker products and reports higher accuracy with up to 27x fewer parameters than QLoRA on 4-bit LLaMA models.

  2. Time Series Foundation Models for Multivariate Financial Time Series Forecasting

    q-fin.GN 2025-07 reject novelty 6.0 of 10

    Pretrained TTM shows large transfer and sample-efficiency gains in three financial forecasting tasks relative to training from scratch, but methodological flaws including possible look-ahead bias weaken the quantitati...

  3. Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization

    cs.LG 2025-07 conditional novelty 6.0 of 10

    SubLoRA projects the Hessian of the fine-tuning loss onto a submodular quadratic objective and uses greedy selection to automatically allocate LoRA ranks under a budget.

  4. Noise Consistency Regularization for Improved Subject-Driven Image Synthesis

    cs.GR 2025-06 conditional novelty 6.0 of 10

    Adding consistency-to-pretrained and multiplicative-noise consistency losses to fine-tuning improves subject identity and background diversity over DreamBooth on a 30-subject benchmark.

  5. SingLoRA: Low Rank Adaptation Using a Single Matrix

    cs.AI 2025-07 conditional novelty 5.0 of 10

    SingLoRA replaces LoRA's two matrices A and B with one matrix A and the symmetric update AA^T, cutting adapter parameters roughly in half while claiming more stable fine-tuning.

  6. Parameter-Efficient Fine-Tuning of 3D DDPM for MRI Image Generation Using Tensor Networks

    eess.IV 2025-07 conditional novelty 4.0 of 10

    TenVOO represents 3D convolution weight updates as tensor networks, fine-tuning a brain MRI DDPM with only 0.3% of full trainable parameters while achieving competitive or better structural similarity on ADNI, PPMI, a...

  7. Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning

    cs.LG 2025-05 reject novelty 4.0 of 10

    The paper claims PEFT is a strict, less robust, lower-capacity subset of full fine-tuning, but the mathematical proofs contain load-bearing errors and the experiments, while suggestive, cannot repair them.

  8. $\mu$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Test-time Wanda pruning, reframed as a mixture of micro-experts, adapts the sparse weight mask to each prompt and improves perplexity and VQA accuracy over static pruning baselines.

  9. Parameter-Efficient Fine-Tuning of Multispectral Foundation Models for Hyperspectral Image Classification

    cs.CV 2025-05 conditional novelty 4.0 of 10

    KronA+ fine-tunes SpectralGPT for hyperspectral image classification using only 0.056% trainable parameters and reaches accuracy close to full fine-tuning on five public datasets.

  10. Dual Decomposition of Weights and Singular Value Low Rank Adaptation

    cs.CL 2025-05 conditional novelty 4.0 of 10

    DuDe combines DoRA's magnitude-direction decomposition with PiSSA's SVD-based initialization, reporting consistent but modest accuracy gains over LoRA, DoRA, and PiSSA on commonsense reasoning, GPQA, MMLU, and GSM8K.

  11. Learning Text Styles: A Study on Transfer, Attribution, and Verification

    cs.CL 2025-07 conditional novelty 3.0 of 10

    A thesis compiles published work claiming that lightweight adapters, contrastive disentanglement, and instruction tuning improve text style transfer, authorship attribution, and authorship verification.

  12. Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models

    cs.CL 2025-06 reject novelty 2.0 of 10

    A progressive scheduling trick that updates only the last remaining blocks in later epochs reduces parameter-update counts by about 25% with roughly unchanged GLUE and SQuAD scores.

Pith tools