Pith. sign in

REVIEW 7 cited by

Low-Rank Quantization-Aware Training for LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.06385 v3 pith:I7EEFQHP submitted 2024-06-10 cs.LG cs.AIcs.CL

Yelysei Bondarenko , Riccardo Del Chiaro , Markus Nagel This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords memoryquantizationllmslr-qatlow-rankmodelperformancetraining
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) are omnipresent, however their practical deployment is challenging due to their ever increasing computational and memory demands. Quantization is one of the most effective ways to make them more compute and memory efficient. Quantization-aware training (QAT) methods, generally produce the best quantized performance, however it comes at the cost of potentially long training time and excessive memory usage, making it impractical when applying for LLMs. Inspired by parameter-efficient fine-tuning (PEFT) and low-rank adaptation (LoRA) literature, we propose LR-QAT -- a lightweight and memory-efficient QAT algorithm for LLMs. LR-QAT employs several components to save memory without sacrificing predictive performance: (a) low-rank auxiliary weights that are aware of the quantization grid; (b) a downcasting operator using fixed-point or double-packed integers and (c) checkpointing. Unlike most related work, our method (i) is inference-efficient, leading to no additional overhead compared to traditional PTQ; (ii) can be seen as a general extended pretraining framework, meaning that the resulting model can still be utilized for any downstream task afterwards; (iii) can be applied across a wide range of quantization settings, such as different choices quantization granularity, activation quantization, and seamlessly combined with many PTQ techniques. We apply LR-QAT to LLaMA-1/2/3 and Mistral model families and validate its effectiveness on several downstream tasks. Our method outperforms common post-training quantization (PTQ) approaches and reaches the same model performance as full-model QAT at the fraction of its memory usage. Specifically, we can train a 7B LLM on a single consumer grade GPU with 24GB of memory. Our source code is available at https://github.com/qualcomm-ai-research/LR-QAT

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FPTQuant: Function-Preserving Transforms for LLM Quantization

    cs.LG 2025-06 conditional novelty 7.0 of 10

    FPTQuant introduces function-preserving transforms that make transformer activations amenable to static 4-bit quantization with minimal inference overhead.

  2. Cool-chic 5.0: Faster Encoding and Inter-Feature Entropy Modeling for Overfitted Image Compression

    eess.IV 2026-05 unverdicted novelty 6.0 of 10

    Cool-chic 5.0 delivers 11% lower rate than H.266/VVC and matches modern autoencoders like MLIC++ with 250 times lower decoding complexity through an updated decoder architecture and faster optimization for overfitted codecs.

  3. AutoNeural: Co-Designing Vision-Language Models for NPU Inference

    cs.CL 2025-12 conditional novelty 6.0 of 10

    A NPU-native VLM combining a MobileNet-style encoder with a hybrid Transformer-SSM backbone claims 14x lower latency and 7x lower quantization error over ViT-Transformer baselines, though quantized accuracy is not reported.

  4. LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving

    cs.DC 2025-09 conditional novelty 6.0 of 10

    A W4A8 GEMM kernel using shift-based overflow-safe dequantization and a hardware-scheduled pipeline reports up to 2.9x speedup over prior W4A8 kernels.

  5. RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    RoSTE couples quantization-aware supervised fine-tuning with per-layer Hadamard rotation selection, reducing quantization outliers and improving 4-bit quantized LLM accuracy over SFT-then-PTQ baselines.

  6. Efficient Reasoning on the Edge

    cs.LG 2026-03 accept novelty 5.5 of 10

    LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.

  7. Fine-tuning on simulated data outperforms prompting for agent tone of voice

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Fine-tuning a 1B-parameter LLM on as few as 100 synthetically generated, readability-filtered samples achieved conversational tone more reliably than a verbose system prompt.

Pith tools