REVIEW 7 cited by
Full Parameter Fine-tuning for Large Language Models with Limited Resources
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP) but demand massive GPU resources for training. Lowering the threshold for LLMs training would encourage greater participation from researchers, benefiting both academia and society. While existing approaches have focused on parameter-efficient fine-tuning, which tunes or adds a small number of parameters, few have addressed the challenge of tuning the full parameters of LLMs with limited resources. In this work, we propose a new optimizer, LOw-Memory Optimization (LOMO), which fuses the gradient computation and the parameter update in one step to reduce memory usage. By integrating LOMO with existing memory saving techniques, we reduce memory usage to 10.8% compared to the standard approach (DeepSpeed solution). Consequently, our approach enables the full parameter fine-tuning of a 65B model on a single machine with 8 RTX 3090, each with 24GB memory.Code and data are available at https://github.com/OpenLMLab/LOMO.
Forward citations
Cited by 7 Pith papers
-
EAPO: Enhancing Policy Optimization with On-Demand Expert Assistance
EAPO lets a policy model consult a stronger expert during training, anneals that access to zero, and improves independent math reasoning by about 5 points over self-exploratory RL.
-
SVAgent: AI Agent for Hardware Security Verification Assertion
SVAgent is a prompt-engineering framework that decomposes security requirements into sub-questions to generate SystemVerilog assertions with higher reported accuracy and consistency than direct LLM generation.
-
Low-rank Momentum Factorization for Memory Efficient Training
MoFaSGD keeps a low-rank factored momentum and uses its singular vectors as the update direction, achieving LoRA-level memory with competitive fine-tuning performance, but its convergence proof is flawed.
-
Time Series Foundation Models for Multivariate Financial Time Series Forecasting
Pretrained TTM shows large transfer and sample-efficiency gains in three financial forecasting tasks relative to training from scratch, but methodological flaws including possible look-ahead bias weaken the quantitati...
-
A Large Language Model-Empowered Agent for Reliable and Robust Structural Analysis
An LLM agent that reframes beam analysis as OpenSeesPy code generation reaches over 99 percent reliability on a small benchmark, but chiefly because the prompt contains a near-identical solved example.
-
When Transformers Meet Recommenders: Integrating Self-Attentive Sequential Recommendation with Fine-Tuned LLMs
SASRecLLM, a hybrid of SASRec and a LoRA-fine-tuned LLM, reports modest gains over TALLRec on MovieLens but shows inconsistent and contradictory results on Amazon and in warm-start settings.
-
Structure-Learnable Adapter Fine-Tuning for Parameter-Efficient Large Language Models
A gated-adapter method with a sparsity regularizer is reported to match or slightly beat full fine-tuning on MNLI and BoolQ using 1.4% of parameters, but lacks the experimental detail needed to verify the claim.
Discussion (0). Sign in to comment.