REVIEW 6 cited by
QKD: Quantization-aware Knowledge Distillation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Quantization and Knowledge distillation (KD) methods are widely used to reduce memory and power consumption of deep neural networks (DNNs), especially for resource-constrained edge devices. Although their combination is quite promising to meet these requirements, it may not work as desired. It is mainly because the regularization effect of KD further diminishes the already reduced representation power of a quantized model. To address this short-coming, we propose Quantization-aware Knowledge Distillation (QKD) wherein quantization and KD are care-fully coordinated in three phases. First, Self-studying (SS) phase fine-tunes a quantized low-precision student network without KD to obtain a good initialization. Second, Co-studying (CS) phase tries to train a teacher to make it more quantizaion-friendly and powerful than a fixed teacher. Finally, Tutoring (TU) phase transfers knowledge from the trained teacher to the student. We extensively evaluate our method on ImageNet and CIFAR-10/100 datasets and show an ablation study on networks with both standard and depthwise-separable convolutions. The proposed QKD outperformed existing state-of-the-art methods (e.g., 1.3% improvement on ResNet-18 with W4A4, 2.6% on MobileNetV2 with W4A4). Additionally, QKD could recover the full-precision accuracy at as low as W3A3 quantization on ResNet and W6A6 quantization on MobilenetV2.
Forward citations
Cited by 6 Pith papers
-
Zero-Shot Quantization for Object Detectors using Off-the-Shelf Generative Models
GoodQ uses generative models with information-dense prompting, distribution-aware selection, and teacher-guided noise reduction to achieve SOTA low-bit (W4A4) and extreme-bit (W3A3) zero-shot quantization for object d...
-
Quantizing Recursive Reasoning Models
Per-tensor 4-bit quantization of weight-tied recursive reasoners induces accumulating activation bias that kills accuracy; per-block scaling (MXInt4) restores it.
-
QATMA: Quantization-Aware Training with Multimodal Alignment for Open-Vocabulary Object Detection
A staged curriculum QAT plus text-anchored relational distillation recovers much of the zero-shot detection accuracy lost by 4-bit quantization of YOLO-World.
-
GeLaCo: An Evolutionary Approach to Layer Compression
Evolutionary search over layer-merging configurations, scored by module-wise activation similarity, yields competitive LLM compression and the first size-quality Pareto fronts.
-
Towards One-bit ASR: Extremely Low-bit Conformer Quantization Using Co-training and Stochastic Precision
A co-training and stochastic-precision method achieves 1-bit and 2-bit weight quantization of Conformer ASR with no statistically significant WER increase on several test sets, yielding up to 16.6x compression.
-
Skeleton-to-Image Encoding: Enabling Skeleton Representation Learning via Vision-Pretrained Models
S2I turns skeleton sequences into body-part-arranged image-like tensors, enabling vision-pretrained models for self-supervised skeleton representation learning across heterogeneous formats.
Discussion (0). Continue with ORCID to comment.