Pith. sign in

REVIEW 12 cited by

RPTQ: Reorder-based Post-training Quantization for Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.01089 v4 pith:4JN4UB67 submitted 2023-04-03 cs.CL

Zhihang Yuan , Lin Niu , Jiawei Liu , Wenyu Liu , Xinggang Wang , Yuzhang Shang , Guangyu Sun , Qiang Wu
show 2 more authors
Jiaxiang Wu Bingzhe Wu
This is my paper · ORCID
classification cs.CL
keywords rptqchannelsllmsmemoryquantizationquantizingchallengelanguage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large-scale language models (LLMs) have demonstrated impressive performance, but their deployment presents challenges due to their significant memory usage. This issue can be alleviated through quantization. In this paper, we identify that the challenge in quantizing activations in LLMs arises from varying ranges across channels, rather than solely the presence of outliers. To address this challenge, we introduce a quantization method called RPTQ, which utilizes a reorder-based approach. By rearranging the channels and quantizing them in clusters, RPTQ effectively mitigates the impact of range differences between channels. To minimize the overhead of the reorder operation, we fuse it into the layer norm operation and weights in linear layers. In our experiments, RPTQ achieved a significant breakthrough by utilizing 3-bit activation in LLMs for the first time, resulting in a substantial reduction in memory usage. For instance, quantizing OPT-175b can lead to a memory consumption reduction of up to 80%.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

    cs.LG 2026-07 conditional novelty 6.0 of 10

    MXSens allocates 8-bit precision to the 32 most sensitive columns per layer, 6-bit to moderately sensitive columns, and 4-bit elsewhere in MXINT, improving WikiText-2 perplexity over prior 4-bit LLM quantization methods.

  2. PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

    cs.LG 2026-07 conditional novelty 6.0 of 10

    PolyQ makes fractional-bit CPU LLM inference practical via per-channel bit allocation from {2,3,4,8,16} and compile-time channel-order merging.

  3. BitNet Text Embeddings

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    BITEMBED trains 1.58-bit ternary-weight LLM embedders with contrastive pre-training, supervised distillation, and multi-precision output training, matching FP16 teachers within ~0.6 MMTEB points at ~2x CPU speed.

  4. You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

    cs.CL 2025-11 reject novelty 6.0 of 10

    TAQ estimates per-layer importance from hidden representations and output sensitivity on task calibration data to allocate mixed precision in a training-free PTQ setting, outperforming task-agnostic baselines on accur...

  5. Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Q-resafe restores much of the safety lost in quantized LLMs by distilling the original model's responses through DPO while selectively updating only safety-critical weights.

  6. Efficient Reasoning on the Edge

    cs.LG 2026-03 accept novelty 5.5 of 10

    LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.

  7. Quantize with Confidence? An Empirical Study of Quantization for Code Generation

    cs.SE 2026-07 conditional novelty 5.0 of 10

    4-bit quantization of code LLMs mostly preserves pass@1, but AQLM matches or beats full precision while QuIP# degrades most on complex prompts; sensitivity is model-dependent.

  8. An Empirical Study of OpenPangu Quantization on Ascend NPUs

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    On Ascend NPUs, OpenPangu-7B tolerates 4-bit weight quantization with minor accuracy loss, while OpenPangu-1B degrades sharply on math/code, and 2-bit/binary settings collapse to near-random output.

  9. Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs

    cs.AI 2025-08 conditional novelty 5.0 of 10

    The study introduces TruthfulnessEval and reports that 4-bit quantization preserves simple true/false accuracy, but explicit 'lie' prompts make quantized and full-precision LLMs output falsehoods even when internal pr...

  10. TuneComp: Joint Fine-tuning and Compression for Large Foundation Models

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Jointly fine-tuning and compressing a ViT into pruned low-rank factors with progressive distillation achieves better accuracy for smaller parameter counts than sequential fine-tune-then-compress pipelines on CIFAR-100.

  11. Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Squeeze10-LLM is a staged mixed-precision post-training quantization method that binarizes 80% of LLM weights, keeps 20% at 4 bits, and reports strong sub-2-bit accuracy gains over PB-LLM.

  12. NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics

    cs.LG 2025-05 conditional novelty 4.0 of 10

    NQKV is a block-wise 4-bit KV cache quantizer that uses a normal-distribution lookup table to let OPT models run with larger batches or longer contexts at negligible measured accuracy cost.

Pith tools