Pith. sign in

REVIEW 5 cited by

GPTVQ: The Blessing of Dimensionality for LLM Quantization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.15319 v2 pith:62XTJQZ7 submitted 2024-02-23 cs.LG cs.CL

Mart van Baalen , Andrey Kuzmin , Ivan Koryakovskiy , Markus Nagel , Peter Couperus , Cedric Bastoul , Eric Mahurin , Tijmen Blankevoort
show 1 more author
Paul Whatmough
This is my paper · ORCID
classification cs.LGcs.CL
keywords quantizationmethodgptvqaccuracycodebooksdimensionalityefficientimproved
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this work we show that the size versus accuracy trade-off of neural network quantization can be significantly improved by increasing the quantization dimensionality. We propose the GPTVQ method, a new fast method for post-training vector quantization (VQ) that scales well to Large Language Models (LLMs). Our method interleaves quantization of one or more columns with updates to the remaining unquantized weights, using information from the Hessian of the per-layer output reconstruction MSE. Quantization codebooks are initialized using an efficient data-aware version of the EM algorithm. The codebooks are then updated, and further compressed by using integer quantization and SVD-based compression. GPTVQ establishes a new state-of-the art in the size vs accuracy trade-offs on a wide range of LLMs such as Llama-v2 and Mistral. Furthermore, our method is efficient: on a single H100 it takes between 3 and 11 hours to process a Llamav2-70B model, depending on quantization setting. Lastly, with on-device timings for VQ decompression on a mobile CPU we show that VQ leads to improved latency compared to using a 4-bit integer format.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Leech Lattice Vector Quantization for Efficient LLM Compression

    cs.LG 2026-03 conditional novelty 6.5 of 10

    Leech-lattice vector quantization with codebook-free indexing and shape-gain search reaches state-of-the-art ~2-bit post-training LLM compression and reduces reliance on Hadamard rotations.

  2. A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inference

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Motion-aware vector quantization plus centroid-reuse GEMM and a custom accelerator yield multi-fold VLA speedups over GPU and prior accelerators with ~2.5% average success-rate drop.

  3. CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs

    cs.LG 2025-07 conditional novelty 6.0 of 10

    CCQ compresses LLMs to 2.0-2.75 bits per weight using convolutional codes and bit-shift decoding, shrinking 671B-parameter models to under 200GB.

  4. PCDVQ: Enhancing Vector Quantization for Large Language Models via Polar Coordinate Decoupling

    cs.LG 2025-06 conditional novelty 6.0 of 10

    PCDVQ compresses LLM weights to 2 bits by quantizing vector directions and magnitudes separately with distribution-matched codebooks, reporting modest zero-shot accuracy gains over prior vector quantization baselines.

  5. LCD: Advancing Extreme Low-Bit Clustering for Large Language Models via Knowledge Distillation

    cs.LG 2025-05 reject novelty 5.0 of 10

    LCD clusters LLM weights into tiny codebooks under a Hessian-guided objective and uses lookup-table inference to reach 2-3 bits, with reported speedups up to 6.2x.

Pith tools