REVIEW 6 cited by
I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Post-training quantization (PTQ) serves as a potent technique to accelerate the inference of large language models (LLMs). Nonetheless, existing works still necessitate a considerable number of floating-point (FP) operations during inference, including additional quantization and de-quantization, as well as non-linear operators such as RMSNorm and Softmax. This limitation hinders the deployment of LLMs on the edge and cloud devices. In this paper, we identify the primary obstacle to integer-only quantization for LLMs lies in the large fluctuation of activations across channels and tokens in both linear and non-linear operations. To address this issue, we propose I-LLM, a novel integer-only fully-quantized PTQ framework tailored for LLMs. Specifically, (1) we develop Fully-Smooth Block-Reconstruction (FSBR) to aggressively smooth inter-channel variations of all activations and weights. (2) to alleviate degradation caused by inter-token variations, we introduce a novel approach called Dynamic Integer-only MatMul (DI-MatMul). This method enables dynamic quantization in full-integer matrix multiplication by dynamically quantizing the input and outputs with integer-only operations. (3) we design DI-ClippedSoftmax, DI-Exp, and DI-Normalization, which utilize bit shift to execute non-linear operators efficiently while maintaining accuracy. The experiment shows that our I-LLM achieves comparable accuracy to the FP baseline and outperforms non-integer quantization methods. For example, I-LLM can operate at W4A4 with negligible loss of accuracy. To our knowledge, we are the first to bridge the gap between integer-only quantization and LLMs. We've published our code on anonymous.4open.science, aiming to contribute to the advancement of this field.
Forward citations
Cited by 6 Pith papers
-
When Can Depth Replace Precision? A Resource Theory of Quantized Neural Computation
For a fixed low-bit residual library, the distance to the closed relaxed reachable set is an exact structural floor that pure depth approaches at O(1/D), while write-back arithmetic can reverse the gain and accuracy m...
-
IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference
A lookup-table-based integer softmax replaces the floating-point softmax in quantized transformer attention, delivering up to 3.7x speedup and 61% energy reduction on ARMv8 CPUs with roughly unchanged accuracy.
-
LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression
Loss-aware rank allocation plus residual-stream output correction yields substantially lower WikiText-2 perplexity than prior SVD LLM compressors at 60% compression.
-
Break Through the Compression Bottleneck: From Theory to Practice
The paper asserts a first proof that low-rank decomposition and quantization are non-orthogonal tools for LLM compression, recommends low-rank-first ordering, and adds a diagonal scaling fix (DAM) that reduces the com...
-
MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantization
MQuant quantizes multimodal LLMs to W4A8 using per-modality static scales, token reordering, and Hadamard-outlier suppression, claiming near-floating-point accuracy and up to 30% latency reduction on five MLLMs.
-
A Survey: Towards Privacy and Security in Mobile Large Language Models
A survey of privacy and security challenges for mobile large language models, summarizing known attack types and defenses without introducing new results.
Discussion (0). Continue with ORCID to comment.