Pith. sign in

REVIEW 10 cited by

BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.10631 v1 pith:MNNJ2Z7I submitted 2024-02-16 cs.CL

Dayou Du , Yijia Zhang , Shijie Cao , Jiaqi Guo , Ting Cao , Xiaowen Chu , Ningyi Xu This is my paper · ORCID
classification cs.CL
keywords bitdistillerlanguagellmsperformancequantizationself-distillationsub-4-bittraining
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The upscaling of Large Language Models (LLMs) has yielded impressive advances in natural language processing, yet it also poses significant deployment challenges. Weight quantization has emerged as a widely embraced solution to reduce memory and computational demands. This paper introduces BitDistiller, a framework that synergizes Quantization-Aware Training (QAT) with Knowledge Distillation (KD) to boost the performance of LLMs at ultra-low precisions (sub-4-bit). Specifically, BitDistiller first incorporates a tailored asymmetric quantization and clipping technique to maximally preserve the fidelity of quantized weights, and then proposes a novel Confidence-Aware Kullback-Leibler Divergence (CAKLD) objective, which is employed in a self-distillation manner to enable faster convergence and superior model performance. Empirical evaluations demonstrate that BitDistiller significantly surpasses existing methods in both 3-bit and 2-bit configurations on general language understanding and complex reasoning benchmarks. Notably, BitDistiller is shown to be more cost-effective, demanding fewer data and training resources. The code is available at https://github.com/DD-DuDa/BitDistiller.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FPTQuant: Function-Preserving Transforms for LLM Quantization

    cs.LG 2025-06 conditional novelty 7.0 of 10

    FPTQuant introduces function-preserving transforms that make transformer activations amenable to static 4-bit quantization with minimal inference overhead.

  2. Reliability Scaling Laws for Quantized Large Language Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Reliability of quantized LLMs peaks nonlinearly at 4-bit precision under fixed total model bits, while accuracy scales monotonically, and quantization can improve robustness to natural perturbations.

  3. SiLQ: Simple Large Language Model Quantization-Aware Training

    cs.LG 2025-07 conditional novelty 6.0 of 10

    SiLQ fine-tunes 8B-parameter LLMs with quantized weights, activations, and cache for a small fraction of extra training tokens, matching or beating leading post-training quantization methods.

  4. Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Q-resafe restores much of the safety lost in quantized LLMs by distilling the original model's responses through DPO while selectively updating only safety-critical weights.

  5. Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMs

    cs.LG 2025-06 conditional novelty 6.0 of 10

    UPQ, a progressive FP16-to-INT4-to-INT2 pipeline with teacher-student distillation, is the first to quantize open-source instruction-tuned LLMs to 2-bit without proprietary post-training data.

  6. RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    RoSTE couples quantization-aware supervised fine-tuning with per-layer Hadamard rotation selection, reducing quantization outliers and improving 4-bit quantized LLM accuracy over SFT-then-PTQ baselines.

  7. Efficient Reasoning on the Edge

    cs.LG 2026-03 accept novelty 5.5 of 10

    LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.

  8. Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs

    cs.AI 2025-08 conditional novelty 5.0 of 10

    The study introduces TruthfulnessEval and reports that 4-bit quantization preserves simple true/false accuracy, but explicit 'lie' prompts make quantized and full-precision LLMs output falsehoods even when internal pr...

  9. BiVM: Accurate Binarized Neural Network for Efficient Video Matting

    cs.CV 2025-07 conditional novelty 5.0 of 10

    BiVM is a 1-bit binarized video matting network that beats prior binarized methods on accuracy and efficiency, with 11.82 MAD on VideoMatte240K versus 28.49 for ReActNet-binarized RVM.

  10. LCD: Advancing Extreme Low-Bit Clustering for Large Language Models via Knowledge Distillation

    cs.LG 2025-05 reject novelty 5.0 of 10

    LCD clusters LLM weights into tiny codebooks under a Hessian-guided objective and uses lookup-table inference to reach 2-3 bits, with reported speedups up to 6.2x.

Pith tools