Pith. sign in

REVIEW 4 cited by

ReLoRA: High-Rank Training Through Low-Rank Updates

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2307.05695 v4 pith:WZEOCBI7 submitted 2023-07-11 cs.CL cs.LG

Vladislav Lialin , Namrata Shivagunde , Sherin Muckatira , Anna Rumshisky This is my paper · ORCID
classification cs.CLcs.LG
keywords trainingreloranetworkshigh-ranklargelow-rankmodelsneural
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Despite the dominance and effectiveness of scaling, resulting in large networks with hundreds of billions of parameters, the necessity to train overparameterized models remains poorly understood, while training costs grow exponentially. In this paper, we explore parameter-efficient training techniques as an approach to training large neural networks. We introduce a novel method called ReLoRA, which utilizes low-rank updates to train high-rank networks. We apply ReLoRA to training transformer language models with up to 1.3B parameters and demonstrate comparable performance to regular neural network training. ReLoRA saves up to 5.5Gb of RAM per GPU and improves training speed by 9-40% depending on the model size and hardware setup. Our findings show the potential of parameter-efficient techniques for large-scale pre-training.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SLORR: Simple and Efficient In-Training Low-Rank Regularization

    cs.LG 2026-07 accept novelty 6.0 of 10

    A stateless, SVD-free regularizer approximates polar factors to induce low-rank weight structure during training, enabling better post-training compression of vision models and LLMs at under 8% overhead.

  2. SALAAD: Sparse And Low-Rank Adaptation via ADMM for Large Language Model Inference

    cs.LG 2026-02 conditional novelty 6.0 of 10

    SALAAD uses an ADMM-style optimizer and an adaptive controller to force LLM weights into sparse-plus-low-rank form during pretraining, so one checkpoint can be elastically deployed at many parameter budgets.

  3. Continual Gradient Low-Rank Projection Fine-Tuning for LLMs

    cs.LG 2025-07 conditional novelty 5.0 of 10

    GORP jointly trains LoRA and full-rank parameters inside a low-rank gradient subspace built from Adam first moments, reporting higher average accuracy and lower forgetting than O-LoRA and N-LoRA on LLM continual learn...

  4. Geometrically Principled Randomized Optimization for Efficient LLM Training

    cs.LG 2025-10 reject novelty 4.0 of 10

    Randomized Grassmannian subspace updates, combined with Adam-state alignment and residual recovery, produce small evaluation-loss gains over prior low-rank LLM training methods.

Pith tools