Pith. sign in

REVIEW 5 cited by

BinaryBERT: Pushing the Limit of BERT Quantization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2012.15701 v2 pith:ARICQWDR submitted 2020-12-31 cs.CL

Haoli Bai , Wei Zhang , Lu Hou , Lifeng Shang , Jing Jin , Xin Jiang , Qun Liu , Michael Lyu
show 1 more author
Irwin King
This is my paper · ORCID
classification cs.CL
keywords binarybertternarybertmodelquantizationsplittingbinarycompression
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular solution. In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization. We find that a binary BERT is hard to be trained directly than a ternary counterpart due to its complex and irregular loss landscape. Therefore, we propose ternary weight splitting, which initializes BinaryBERT by equivalently splitting from a half-sized ternary network. The binary model thus inherits the good performance of the ternary one, and can be further enhanced by fine-tuning the new architecture after splitting. Empirical results show that our BinaryBERT has only a slight performance drop compared with the full-precision model while being 24x smaller, achieving the state-of-the-art compression results on the GLUE and SQuAD benchmarks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 45 citations worldwide. Full citation record

  1. A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inference

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Motion-aware vector quantization plus centroid-reuse GEMM and a custom accelerator yield multi-fold VLA speedups over GPU and prior accelerators with ~2.5% average success-rate drop.

  2. LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits

    cs.LG 2025-02 conditional novelty 6.0 of 10

    LowRA enables LoRA fine-tuning with base weights at 1.15 to 4 bits per parameter, outperforming QLoRA and LoftQ at equal bit widths and matching their accuracy at lower bit widths.

  3. Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Structural pruning with finetuning plus hidden-state distillation recovers most performance in multimodal LLMs, with 5% of training data sufficient at moderate compression levels.

  4. ACE: Exploring Activation Cosine Similarity and Variance for Accurate and Calibration-Efficient LLM Pruning

    cs.LG 2025-05 conditional novelty 5.0 of 10

    ACE adds activation cosine-similarity and activation-variance terms to the per-weight importance score, and reports better perplexity and lower pruning time than Wanda and RIA on LLaMA, LLaMA-2, and OPT.

  5. EGGS-PTP: An Expander-Graph Guided Structured Post-training Pruning Method for Large Language Models

    cs.LG 2025-08 conditional novelty 4.0 of 10

    EGGS-PTP adds a connectivity-preserving diagonal selection to RIA-style importance pruning, achieving slightly better perplexity under N:M sparsity.

Pith tools