Pith. sign in

REVIEW 15 cited by

VeRA: Vector-based Random Matrix Adaptation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.11454 v2 pith:7KLEXAZO submitted 2023-10-17 cs.CL

Dawid J. Kopiczko , Tijmen Blankevoort , Yuki M. Asano This is my paper · ORCID
classification cs.CL
keywords modelsadaptationlanguageloralow-rankmatrixnumberparameters
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Low-rank adapation (LoRA) is a popular method that reduces the number of trainable parameters when finetuning large language models, but still faces acute storage challenges when scaling to even larger models or deploying numerous per-user or per-task adapted models. In this work, we present Vector-based Random Matrix Adaptation (VeRA), which significantly reduces the number of trainable parameters compared to LoRA, yet maintains the same performance. It achieves this by using a single pair of low-rank matrices shared across all layers and learning small scaling vectors instead. We demonstrate its effectiveness on the GLUE and E2E benchmarks, image classification tasks, and show its application in instruction-tuning of 7B and 13B language models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TerraNova: A Foundation Model for the Anthropocene

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A multi-geometry model, trained on 512 gridded Earth-system fields and 512 national indicators, couples physical and societal data in one latent space and supports uncertainty-aware reconstruction and downscaling.

  2. RSRA: Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation

    cs.CV 2026-07 reject novelty 6.0 of 10

    Training-free LoRA rank allocation via virtual low-rank noise probing that scores modules by Fréchet distance and effective rank of activation shifts outperforms prior heuristic and training-based allocators on standa...

  3. Low-rank Momentum Factorization for Memory Efficient Training

    cs.LG 2025-07 reject novelty 6.0 of 10

    MoFaSGD keeps a low-rank factored momentum and uses its singular vectors as the update direction, achieving LoRA-level memory with competitive fine-tuning performance, but its convergence proof is flawed.

  4. PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    PLoP selects LoRA adapter placement by ranking normalized feature norms and placing adapters on the lowest-scoring module types, using only forward passes.

  5. Noise Consistency Regularization for Improved Subject-Driven Image Synthesis

    cs.GR 2025-06 conditional novelty 6.0 of 10

    Adding consistency-to-pretrained and multiplicative-noise consistency losses to fine-tuning improves subject identity and background diversity over DreamBooth on a 30-subject benchmark.

  6. HRP: High-Rank Preheating for Superior LoRA Initialization

    cs.LG 2025-02 conditional novelty 6.0 of 10

    HRP initializes LoRA with the top singular vectors of a briefly preheated high-rank adapter, improving fine-tuning results over random initialization in experiments.

  7. Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning

    cs.LG 2026-07 reject novelty 5.0 of 10

    A retraction-free Stiefel manifold optimization algorithm with a fixed penalty parameter is proposed and applied to LoRA fine-tuning, claiming faster convergence and better downstream performance.

  8. Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Applying mHC as a PEFT method shows that learned residual mixing is unnecessary — even harmful — in finetuning, and mHC+LoRA combinations give small task-dependent gains.

  9. L1RA: Dynamic Rank Assignment in LoRA Fine-Tuning

    cs.CL 2025-09 conditional novelty 5.0 of 10

    L1RA uses L1 regularization to prune and reallocate LoRA ranks across adapters during fine-tuning, achieving small perplexity gains on two 7B to 8B models at nearly the same training time.

  10. FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FLoE uses Fisher information to pick the transformer layers that matter and a Bayesian optimizer to set LoRA rank, cutting trainable parameters while keeping or improving accuracy.

  11. UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    UORA is a LoRA/VeRA-style PEFT method that selectively reinitializes low-magnitude rows and columns of frozen random matrices, reaching LoRA-comparable performance with far fewer trainable parameters.

  12. GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation

    cs.LG 2025-08 conditional novelty 4.0 of 10

    GEM selects fine-tuning parameters by gradient-to-weight ratio and distributes the budget by layer entropy, reaching 0.1% parameter updates with small accuracy gains on several NLP tasks.

  13. Efficient Adaptation of Pre-trained Vision Transformer underpinned by Approximately Orthogonal Fine-Tuning Strategy

    cs.CV 2025-07 conditional novelty 4.0 of 10

    AOFT generates approximately orthogonal down/up projection matrices from a single learnable vector, matching the observed near-orthogonality of pre-trained ViT weights, and reports competitive image-classification per...

  14. MAP: Revisiting Weight Decomposition for Low-Rank Adaptation

    cs.LG 2025-05 conditional novelty 4.0 of 10

    MAP decouples a weight matrix's direction and magnitude by normalizing the whole matrix and the low-rank update by their Frobenius norms and scaling each with a learnable scalar.

  15. The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

    cs.AI 2026-06 unverdicted novelty 2.0 of 10

    A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.

Pith tools