REVIEW 30 cited by
Learned Step Size Quantization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Deep networks run with low precision operations at inference time offer power and space advantages over high precision alternatives, but need to overcome the challenge of maintaining high accuracy as precision decreases. Here, we present a method for training such networks, Learned Step Size Quantization, that achieves the highest accuracy to date on the ImageNet dataset when using models, from a variety of architectures, with weights and activations quantized to 2-, 3- or 4-bits of precision, and that can train 3-bit models that reach full precision baseline accuracy. Our approach builds upon existing methods for learning weights in quantized networks by improving how the quantizer itself is configured. Specifically, we introduce a novel means to estimate and scale the task loss gradient at each weight and activation layer's quantizer step size, such that it can be learned in conjunction with other network parameters. This approach works using different levels of precision as needed for a given system and requires only a simple modification of existing training code.
Forward citations
Cited by 30 Pith papers
-
Zero-Shot Quantization for Object Detectors using Off-the-Shelf Generative Models
GoodQ uses generative models with information-dense prompting, distribution-aware selection, and teacher-guided noise reduction to achieve SOTA low-bit (W4A4) and extreme-bit (W3A3) zero-shot quantization for object d...
-
When Bits Break Recourse: Counterfactual-Faithful Quantization
Quantization can preserve accuracy while invalidating algorithmic recourse; CFQ trains the quantized model at teacher recourse points and preserves counterfactual validity and recourse cost.
-
Quantization Meets OOD: Generalizable Quantization-aware Training from a Flatness Perspective
Quantization-aware training degrades out-of-distribution accuracy, and a flatness-aware method with gradient-disorder freezing, FQAT, partially recovers it.
-
CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training
CAGE, a curvature-aware correction that adds the quantization error to the gradient, reduces loss in low-bit quantization-aware training, letting 3-bit CAGE-trained models match 4-bit baseline-trained models.
-
MSQ: Memory-Efficient Bit Sparsification Quantization
MSQ computes and prunes least significant bits of weights directly from the full-precision parameters, cutting training memory and time for mixed-precision quantization compared with bit-splitting methods.
-
Task-Specific Zero-shot Quantization-Aware Training for Object Detection
A zero-shot quantization-aware training method for object detectors that synthesizes task-specific images with bounding-box labels via adaptive label sampling, then distills task-specific knowledge into the quantized network.
-
DMQ: Dissecting Outliers of Diffusion Models for Post-Training Quantization
A post-training quantization method that combines learned channel scaling and power-of-two scaling keeps diffusion image quality high at 4-bit weight, 6-bit activation precision.
-
Post-Training Quantization of Generative and Discriminative LSTM Text Classifiers: A Study of Calibration, Class Balance, and Robustness
Generative LSTM classifiers under post-training quantization are far more sensitive than discriminative ones to calibration data class balance and input noise, especially at 3- to 5-bit widths.
-
Is Quantization a Deal-breaker? Empirical Insights from Large Code Models
4-bit AWQ quantization of CodeLlama and DeepSeek-Coder largely preserves static-analysis measures of code quality on Java and Python tasks, with small readability and maintainability shifts.
-
MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation
MPQ-DMv2 adds binary residual quantization, temporal relation distillation, and SVD-initialized LoRA to mixed-precision quantization, improving low-bit diffusion model generation quality.
-
TruncQuant: Truncation-Ready Quantization for DNNs with Flexible Weight Bit Precision
TruncQuant uses a floor-based quantizer with 2^n scaling instead of rounding with 2^n-1, so truncating high-precision weights via bit-shift exactly matches direct low-precision quantization, recovering accuracy lost b...
-
Unified Scaling Laws for Compressed Representations
A representation capacity derived from Gaussian fitting error predicts the training efficiency of sparse, quantized, and hybrid compressed models, and this capacity approximately multiplies across combined compression types.
-
Flexible Mixed Precision Quantization for Learned Image Compression
A rate-distortion sensitivity criterion assigns per-layer bit-widths, yielding about 1 to 2 percent BD-Rate improvement over 8-bit fixed-precision quantization at matched model size for learned image compression.
-
Frequency Composition for Compressed and Domain-Adaptive Neural Networks
Training quantized models on low-frequency images plus frequency-aware batch normalization at test time improves both compression and domain-shift robustness.
-
Is Quantum Optimization Ready? An Effort Towards Neural Network Compression using Adiabatic Quantum Computing
Joint pruning-quantization of small convolutional networks is cast as a QUBO and solved on D-Wave and Fujitsu annealers with under 1% accuracy drop at high compression rates.
-
Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference
DPQ is a post-training quantization algorithm that stores weights in INT4, computes in FP8, and uses Hessian-based group-aware reordering to keep accuracy near the full-precision model.
-
Scaling Law for Quantization-Aware Training
A unified QAT scaling law predicts 4-bit quantization error from model size, training tokens, and group size, showing activation outliers in the FC2 layer are the main W4A4 bottleneck.
-
Quantize with Confidence? An Empirical Study of Quantization for Code Generation
4-bit quantization of code LLMs mostly preserves pass@1, but AQLM matches or beats full precision while QuIP# degrades most on complex prompts; sensitivity is model-dependent.
-
QuantV2X: A Fully Quantized Multi-Agent System for Cooperative Perception
QuantV2X shows that a fully quantized multi-agent fusion system reduces end-to-end latency by 3.2x and improves system-level mAP30 by 9.5 over a full-precision system on the V2X-Real dataset.
-
DFQ-ViT: Data-Free Quantization for Vision Transformers without Fine-tuning
DFQ-ViT combines easy-to-hard synthetic sample generation with activation correction to quantize vision transformers without data or fine-tuning, outperforming PSAQ-ViT and roughly matching real-data calibration.
-
PQCAD-DM: Progressive Quantization and Calibration-Assisted Distillation for Extremely Efficient Diffusion Model
PQCAD-DM combines progressive two-stage quantization with calibration-assisted distillation to halve the sampling steps of a quantized diffusion model while keeping FID scores close to baseline.
-
Compression Aware Certified Training
CACTUS trains a single network on pruned and weight-perturbed copies of itself, beating prior certified-training baselines on compressed MNIST and CIFAR-10 models.
-
GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers
A two-stage 'activation-first, weights-later' quantization method that reaches competitive 4-bit ViT accuracy with about one epoch of training.
-
ScalableHD: Scalable and High-Throughput Hyperdimensional Computing Inference on Multi-Core CPUs
ScalableHD speeds up hyperdimensional computing inference on multi-core CPUs by up to 10x (and up to ~18x in tables) over the TorchHD baseline while keeping the same model and accuracy.
-
Opt.Gear Technical Report
Opt.Gear is a family of efficient on-device language models using a ConvKV-gated mixer with sparse attention, trained on 0.5T tokens without distillation, claiming up to 4.9x NPU speedups and 20 TPS on a Cortex-M7.
-
Local Stability and Gaussian Smoothing of Quantized Neural Networks
Gaussian averaging of quantized neural models yields C∞ surrogates with a local dimension-dependent error bound under a bounded oscillation condition, plus a CLT-based Gaussian envelope for binary perceptrons.
-
TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantization
TinyMusician distills MusicGen and applies hand-picked mixed-precision quantization to make a 1.04 GB on-device music generator, but the headline '93% quality, 55% smaller' claims conflict with the paper's own tables.
-
Progressive Element-wise Gradient Estimation for Neural Network Quantization
PEGE replaces the straight-through estimator with a curriculum-driven blend of quantized and full-precision weights plus an additive discretization-error correction, reporting small accuracy gains on low-bit CNNs.
-
Structured Pruning and Quantization for Learned Image Compression
A rate-distortion-aware neural architecture search plus joint pruning and 8-bit quantization reduces learned image compression model size by up to about 4.9x with small rate-distortion penalty.
-
Streaming Chain
Prune then INT8 QAT then KD yields a better measured accuracy–size–latency trade-off on CIFAR CNNs than any single stage, and stage order itself changes accuracy.
Discussion (0). Sign in to comment.