REVIEW 19 cited by
8-bit Optimizers via Block-wise Quantization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Stateful optimizers maintain gradient statistics over time, e.g., the exponentially smoothed sum (SGD with momentum) or squared sum (Adam) of past gradient values. This state can be used to accelerate optimization compared to plain stochastic gradient descent but uses memory that might otherwise be allocated to model parameters, thereby limiting the maximum size of models trained in practice. In this paper, we develop the first optimizers that use 8-bit statistics while maintaining the performance levels of using 32-bit optimizer states. To overcome the resulting computational, quantization, and stability challenges, we develop block-wise dynamic quantization. Block-wise quantization divides input tensors into smaller blocks that are independently quantized. Each block is processed in parallel across cores, yielding faster optimization and high precision quantization. To maintain stability and performance, we combine block-wise quantization with two additional changes: (1) dynamic quantization, a form of non-linear optimization that is precise for both large and small magnitude values, and (2) a stable embedding layer to reduce gradient variance that comes from the highly non-uniform distribution of input tokens in language models. As a result, our 8-bit optimizers maintain 32-bit performance with a small fraction of the memory footprint on a range of tasks, including 1.5B parameter language modeling, GLUE finetuning, ImageNet classification, WMT'14 machine translation, MoCo v2 contrastive ImageNet pretraining+finetuning, and RoBERTa pretraining, without changes to the original optimizer hyperparameters. We open-source our 8-bit optimizers as a drop-in replacement that only requires a two-line code change.
Forward citations
Cited by 19 Pith papers
-
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
Full-stack NVFP4 LLM pretraining is stable with a 1.47% loss gap to BF16 via LoRA-SVD projections, transformed AdamW/Root optimizers, and mixed-precision attention.
-
Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
Constraining transformer projection weights to a shared low-rank subspace reportedly enables near-lossless compression of pipeline-parallel communication, matching centralized convergence at 80Mbps bandwidth.
-
Gefen: Optimized Stochastic Optimizer
Gefen automatically groups parameters for shared second moments and exact-DP-quantizes first moments, matching AdamW quality at roughly 1/8 the optimizer memory.
-
FBLayout: Optimizing Memory Layout for Efficient LLM Finetuning on Mobile GPUs
A tile-based memory layout for mobile GPUs that unifies forward and backward data access, eliminating most transpose/reshape overhead and speeding LLM fine-tuning 2.2–5.7× in the paper's measurements.
-
OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.
-
Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers
Benchmarking on four edge platform configurations shows hardware accelerators improve LLM inference efficiency and reveals trade-offs in power use, device size, and token throughput for constrained deployments.
-
Droplet3D: Commonsense Priors from Videos Facilitate 3D Generation
A video diffusion backbone fine-tuned on 4M densely captioned 360-degree renderings generates spatially consistent multi-view images for 3D assets from image plus detailed text input.
-
Is Quantization a Deal-breaker? Empirical Insights from Large Code Models
4-bit AWQ quantization of CodeLlama and DeepSeek-Coder largely preserves static-analysis measures of code quality on Java and Python tasks, with small readability and maintainability shifts.
-
Low-rank Momentum Factorization for Memory Efficient Training
MoFaSGD keeps a low-rank factored momentum and uses its singular vectors as the update direction, achieving LoRA-level memory with competitive fine-tuning performance, but its convergence proof is flawed.
-
Efficient Multilingual Neural Machine Translation via Corpus-Driven Vocabulary Pruning: An English-Arabic Case Study
Pruning multilingual NMT vocabularies to corpus-relevant tokens plus fine-tuning cuts memory by about 60% and matches or beats a dedicated English-Arabic model on COMET and TER.
-
Reasoning Before Translation: Enhancing Legal Machine Translation with Structured Reasoning
On Swiss legal translation, reinforcement learning with a ChrF reward improves small open models more than supervised fine-tuning, but frontier reasoning models still score higher.
-
RAGAL: A Frugal, Fully Local Retrieval-Augmented Assistant for Technical Support at a Government Agency
In a fully local Romanian RAG assistant, retrieval engineering and a 72-minute embedder fine-tune lifted ticket recall@10 from 0.663 to 0.850, while per-domain evaluation exposed and fixed silent document-domain regression.
-
When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models
Quantized (INT8/INT4) LLMs can outperform FP16 in later-task forward accuracy and retention during continual learning, though single-seed runs leave the effect unquantified.
-
Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendation
Quantizing LLMs to 4 or 8 bits cuts GPU memory by up to 75% with generally small performance changes across eight biomedical NLP benchmarks.
-
A MISMATCHED Benchmark for Scientific Natural Language Inference
MISMATCHED is a new out-of-domain benchmark for scientific NLI spanning three non-CS domains, with best baselines at 78.17% Macro F1 and evidence that implicit-relation training helps.
-
Quaff: Quantized Parameter-Efficient Fine-Tuning under Outlier Spatial Stability Hypothesis
Quaff shows that activation outlier channels keep their spatial positions during LLM fine-tuning, and exploits this stability to cut fine-tuning memory and latency with INT8 quantization while matching or beating full...
-
Fine-Tuning Causal LLMs for Text Classification: Embedding-Based vs. Instruction-Based Approaches
Embedding-based QLoRA fine-tuning of causal LLMs matches BERT on single-label patent classification with 10–30x fewer trainable parameters, while instruction-tuning wins on multi-label classification only with ≥100M t...
-
Comparative Evaluation of Prompting and Fine-Tuning for Applying Large Language Models to Grid-Structured Geospatial Data
Fine-tuning a small LLM on 100 self-built geospatial weather examples produced a reported perfect score on a 12-example test set, far above the prompt-only model.
-
Slimming Down LLMs Without Losing Their Minds
QLoRA fine-tuning on Alpaca data drops a 1B model's math accuracy from 33.5% to 3.7% and reduces MMLU-CS accuracy by 13 points, while HellaSwag stays flat.
Discussion (0). Sign in to comment.