BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration

Ahmed F. AbouElhamayed; George A. Constantinides; Marta Andronic; Mohamed S. Abdelfattah; Xilai Dai; Yang Wang; Yuzong Chen

REVIEW 2 major objections 5 minor 1 cited by

BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BitMoD shows that LLM weights can be quantized to 3-bit and 4-bit with better perplexity and accuracy than prior quantization schemes, by giving each small group of weights its own asymmetric floating-point data type and building a…

desk verdict A strong algorithm-hardware co-design paper with a genuinely new per-group data-type idea, held back slightly by special values that are chosen on the same six LLMs used in evaluation. read the letter →

arxiv 2411.11745 v2 pith:J3WSCGPM submitted 2024-11-18 cs.LG cs.AR

Yuzong Chen , Ahmed F. AbouElhamayed , Xilai Dai , Yang Wang , Marta Andronic , George A. Constantinides , Mohamed S. Abdelfattah This is my paper · ORCID

classification cs.LGcs.AR

keywords LLMquantizationweight-onlyper-groupasymmetricfloating-pointformatsbit-serialacceleratorpost-trainingalgorithm-hardwareco-designFP3/FP4datatypes

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the limiting factor in low-precision LLM weight quantization is not the bit-width itself but the data type: standard integer and floating-point formats waste quantization levels on a redundant zero and cannot adapt to the asymmetric, Gaussian-like distribution of weights within a group. BitMoD introduces extended FP3 and FP4 formats that replace the redundant zero with one of four special values, and lets each group of 128 weights pick whichever special value minimizes that group's quantization error. On six LLMs, this fine-grained data type adaptation keeps 4-bit accuracy loss under 0.5% on average and makes 3-bit weights outperform prior quantization schemes in perplexity. The hardware half of the paper shows that a bit-serial processing element can execute these mixed data types with a unified representation, achieving average speedups of 1.69x and 1.48x over the ANT and OliVe accelerators respectively.

What carries the argument

The load-bearing mechanism is the extended asymmetric floating-point data type, derived by repurposing the redundant ±0 of a sign-magnitude format: FP3-ER/EA adds special values ±3 and ±6 to the basic FP3 set {0, ±1, ±2, ±4}, and FP4-ER/EA adds ±5 and ±8 to the basic FP4 set. A fine-grained data type adaptation algorithm (Algorithm 1) quantizes each group of 128 weights against the basic values plus each candidate special value, and selects the special value with the lowest mean-square error, storing the choice in a 2-bit per-group metadata field. On the hardware side, the accelerator decomposes every supported weight format (INT8, INT6, FP4, FP3) into a unified bit-serial representation with sign, exponent, mantissa, and bit-significance fields, and uses a bit-serial dequantization unit that multiplies the accumulated group partial sum by one bit of the 8-bit integer scaling factor per cycle, so per-group dequantization never stalls the dot-product pipeline.

What would settle it

Quantize a held-out set of LLMs (e.g., Mistral, Falcon, or a 30B-parameter model) at 3-bit with BitMoD's fixed special values {±3, ±6} and compare perplexity against per-group asymmetric integer quantization; if the perplexity advantage over INT3-Asym shrinks below the reported margin or reverses, the special-value selection is overfit to the benchmark models. A more direct test is to measure normalized per-group quantization error across a broad model zoo and check whether ±3 and ±6 still minimize the average error.

Watch

Extended reading notes

Core claim

The paper's central claim is that replacing the redundant zero in low-precision floating-point formats with carefully chosen special values turns 3-bit and 4-bit weights into practical formats for LLMs. For FP3 the special values are ±3 (extra resolution inside the FP3 range) and ±6 (extra asymmetry extending the range); for FP4 they are ±5 and ±8. Each weight group is quantized with the basic values plus whichever single special value gives the lowest mean-square error, so different groups in the same tensor can use different data types. The authors report that this per-group adaptation achieves better perplexity than prior data types at 4-bit, and that at 3-bit it keeps perplexity much closer to the FP16 baseline than ANT, OliVe, or Microscaling formats. They further claim that the accompanying bit-serial accelerator, which processes weight values as sign, exponent, mantissa, and bit-significance terms and dequantizes per-group partial sums in a bit-serial manner, delivers 1.69x and 1.48x average speedups over ANT and OliVe under an iso-compute-area constraint.

Load-bearing premise

The four special values were chosen by minimizing per-group quantization error on the six evaluated LLMs, so the claim that they stay near-optimal for other LLMs and datasets is the load-bearing premise.

Editorial extensions

If this is right

At 4-bit weight precision, BitMoD claims average accuracy loss below 0.5% on discriminative tasks, which would make 4-bit weights a drop-in option for edge LLM deployment.
At 3-bit, BitMoD reports better perplexity than prior quantization schemes, implying that sub-4-bit weights can be used for generative tasks without the quality collapse seen with integer formats.
Because the data types are orthogonal to AWQ- and OmniQuant-style optimizations, combining BitMoD with those methods reduces average perplexity loss to below 1.0 at both 3-bit and 4-bit, extending the reach of existing software pipelines.
The bit-serial PE computes four multiply-accumulates in two cycles for FP4/FP3 and three cycles for INT6, giving 1.33x and 2x throughput over an FP16 MAC unit, so the accelerator gains scale with how aggressively weights are quantized.
With INT8 per-group scaling factors and 2-bit special-value metadata, the memory overhead of per-group quantization stays at 10 bits per 128-weight group, which is lower than the 24-bit overhead of standard asymmetric integer formats.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The special-value search procedure could be made per-tensor or per-layer rather than fixed globally, and the paper's hardware already supports reprogrammable special-value registers, so a data-dependent calibration pass is a natural extension.
The same zero-repurposing trick could plausibly extend to 2-bit formats, where the redundant zero occupies an even larger fraction of quantization levels, although accuracy would need to be re-validated.
Because the accelerator's bit-serial decoder is not limited to the four chosen special values, the approach may generalize to future data types without redesign, making the hardware a flexible substrate for quantization research.
The per-group dequantization unit's cost is amortized because the group dot-product takes at least 64 cycles while dequantization takes 8, so the design should scale to smaller group sizes before pipeline stalls appear.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: BitMoD is a well-executed co-design paper. The core idea—repurposing the redundant zero in low-precision floating-point to add either extra resolution (ER) or extra asymmetry (EA) per group—is genuinely new. The hardware side, a unified bit-serial representation plus bit-serial dequantization, is clean and the reported efficiency numbers are plausible. I think the paper deserves a serious referee.

What's actually new: the FP3/FP4 extensions and the per-group adaptation that picks one of four special values for each 128-weight group. The 2-bit encoding overhead is tiny, and the idea of using the redundant zero is elegant—at 3 bits that zero is 12.5% of the quantization levels, so replacing it with something useful is a real lever. The unified bit-serial PE that handles INT8, INT6, FP4, and FP3 with one datapath is also a solid contribution, and the bit-serial dequantization with INT8 scaling factors sensibly avoids a floating-point pipeline.

The paper does most things right. Comparisons against ANT, OliVe, and MX are extended to per-group quantization for fairness, which I appreciate. The ablations (data types, special values) are there. The artifact appendix is real—code, RTL, and simulator are archived—and the RTL synthesis plus cycle simulation is standard practice in this community. Combining BitMoD with AWQ/OmniQuant/SmoothQuant shows the idea is orthogonal to existing optimizations, and those results strengthen the paper.

The soft spot is the one you flagged: the four special values are selected by minimizing per-group quantization error on the exact six LLMs used in the evaluation (Fig. 3, Table IX). The paper is transparent about this, but a leave-one-out or held-out-model check is missing. This is a moderate concern, not a fatal one. The selection criterion is weight MSE, not downstream accuracy, and the chosen values are the best on five of six models, so the risk is that the headline 3-bit advantage is somewhat optimistic for unseen families, not that the mechanism fails. Still, the load-bearing claim at 3-bit would be much stronger with a Mistral or Gemma datapoint. The hardware speedup claims are simulation-based, which is fine; they are also partly contingent on 3-bit remaining acceptable, but the lossless INT6 configuration gives a solid alternative.

Bottom line: the paper is a legitimate HPCA/ISCA-caliber submission. The requested revision is clear: validate the special values on held-out models, and maybe report a couple more model families. I'd cite the data type idea in my own work.

Referee Report

2 major / 5 minor

Summary. The paper proposes BitMoD, a post-training weight-quantization scheme and co-designed accelerator for LLMs. On the algorithm side, BitMoD extends low-precision floating-point formats (FP3/FP4) by replacing the redundant zero with one of four special values (e.g., ±3 and ±6 for FP3) and selecting the best special value per group of 128 weights via 2-bit metadata. This 'fine-grained data type adaptation' is claimed to improve perplexity at 3-bit/4-bit over prior methods and to integrate with AWQ, OmniQuant, and SmoothQuant. On the hardware side, the paper presents a bit-serial PE with a unified representation for INT8/INT6/FP4/FP3 and an on-the-fly bit-serial dequantization unit. Evaluation on six LLMs reports <0.5% average accuracy loss at 4-bit on discriminative tasks, and average speedups of 1.69× and 1.48× over ANT and OliVe under an iso-compute area constraint. The authors provide an open-source artifact and RTL synthesis in TSMC 28nm with cycle-level simulation.

Significance. If the reported gains generalize beyond the evaluation suite, the work is significant: it shows that reusing the zero codepoint in low-precision float formats can push the accuracy-efficiency frontier of LLM weight quantization, and the hardware design is thorough and reproducible (public artifact, RTL synthesis, cycle-accurate simulator, and a clear iso-compute-area methodology). The 3-bit perplexity results are substantially better than the per-group integer baselines on the evaluated models. However, the central algorithmic innovation—the specific choice of special values—is selected on the same six models used for the final evaluation, so the generality of the headline claims is currently unproven. The 3-bit hardware speedup in Fig. 7 is contingent on that generality.

major comments (2)

[Section III-A, Fig. 3, Table IX] The special values ±3 and ±6 for FP3, and ±5 and ±8 for FP4, are chosen by minimizing per-group quantization error on the same six LLMs that are later used to report all perplexity and accuracy results. The only special-value ablation (Table IX) is also performed on a subset of those same models (OPT-1.3B, Phi-2B, Llama-2-7B, Llama-3-8B). The paper explicitly states, 'we have fixed the four special values given that they can minimize the quantization error for the diverse set of LLMs that we evaluate.' Consequently, the reported 3-bit perplexity advantage and the associated 3-bit hardware speedup in Fig. 7 are in-sample. This is load-bearing: if the selected values do not transfer to other LLM families, the algorithmic advantage may vanish and the lossy hardware configuration would need to fall back to 4-bit, eroding the speedup. I request a held-out validation, such as leave-one-model-out selection or evaluation on additional LLMs (e.g., Mistral, Gemma) that were not used in Fig. 3, to demonstrate that the chosen special values remain near-optimal.
[Section III-A, Fig. 3] The procedure for selecting the special values is not described in sufficient detail to assess overfitting risk. The text says 'we determine the two remaining special values that can minimize the quantization error,' but the candidate set, the search strategy, and the stopping criterion are unspecified. Fig. 3 plots only a few candidates, and it is unclear whether the search space included non-integer values or values beyond ±8. Please provide the complete candidate set, the selection criterion applied, and the resulting quantization-error values for all candidates, or prefer a first-principles derivation if one exists. Without this, the claim that the chosen values 'minimize' the error is not reproducible.

minor comments (5)

[Abstract] The phrase 'better perplexity than prior LLM quantization scheme' is vague; please specify which prior scheme(s) are meant, or state the comparison more generally as 'prior low-bit LLM quantization methods.'
[Section II-C, text near Table III] The sentence 'it necessities a floating-point pipeline' should read 'it necessitates a floating-point pipeline.'
[Section IV-B, paragraph on pipeline timing] The argument that bit-serial dequantization 'never stalls' the pipeline assumes that dequantization of one group can overlap with the dot-product of the next group. The paper does not describe double-buffering or a pipeline schedule for the shared column accumulator; a short clarification of the scheduling would help.
[Section V-C, Fig. 7] The 'lossy' configuration uses 3-bit weights for generative tasks, but Table VI shows that some models (e.g., OPT-1.3B, Phi-2B) incur perplexity increases of 8 and 3 points, respectively. It would be useful to state the quality threshold used to declare this configuration 'good', and to report the speedup if a fixed quality target (e.g., within 1 perplexity point of FP16) were enforced.
[Table III and Section V-A] The comparison with ANT and OliVe uses per-group extensions of their data types, which is transparently stated, but the paper should also report the original per-channel results for ANT/OliVe on the same models to let the reader see the effect of the extension.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline 3-bit/4-bit accuracy gains are in-sample: the FP3/FP4 special values are fitted to the exact six LLMs used for evaluation, and no held-out model validation is provided.

fitted input called prediction [Section III-A (FP3 Extension, FP4 Extension), Fig. 3; model evaluation in Section V-B (Tables VI-IX)]
"Fig. 3 shows the normalized per-group quantization error on six LLMs when adding different special values to FP3. ... the special values ± 6 have the lowest quantization error on most LLMs except for OPT-1.3B, and are therefore adopted in BitMoD. ... We conduct experiments to measure the effects of different FP4 special values on the resulting quantization error, which leads to the best FP4-ER and FP4-EA that have special values ±5 and ±8, respectively. ..."

The only free parameters of the proposed data types—the four special values FP3 {±3, ±6} and FP4 {±5, ±8}—are selected by minimizing per-group quantization error on the same six LLMs (OPT-1.3B, Phi-2B, Yi-6B, Llama-2-7B, Llama-2-13B, Llama-3-8B) on which the paper then reports its headline perplexity and accuracy results. The paper explicitly states the values were fixed because they minimize error for 'the diverse set of LLMs that we evaluate,' i.e., the evaluation set is also the selection set. The only special-value ablation (Table IX) is run on a subset of those same models, so it cannot reveal overfitting to the benchmark family.

full rationale

The hardware contribution is largely self-contained: the bit-serial unified representation, the mixed-precision PE with bit-serial dequantization, and the RTL/cycle-level evaluation against FP16, ANT, and OliVe under iso-area constraints do not depend on the fitted special values except through the choice of 'lossy' precision (3-bit generative / 4-bit discriminative), which is itself justified using the same fitted evaluations. The self-citations in the related-work discussion are not load-bearing. However, the central model-quality claim—that BitMoD's new data types enable 3-bit weights with better perplexity than prior schemes—rests on special values that were selected to minimize quantization error on the very six LLMs used for the evaluation. No leave-one-out, held-out LLM family, or external benchmark is provided to show the chosen constants transfer. This is a partial, in-sample circularity: the parameter choice and the headline evaluation share the same data, so the 3-bit/4-bit accuracy numbers are partly forced by selection. Score 6 reflects that the algorithmic headline is fitted on its own evaluation set while the hardware design retains independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central algorithm depends on empirically selected special values and a standard group size; the hardware evaluation depends on standard simulation tools. No new physical entities are introduced.

free parameters (4)

FP3 special values = -3, +3, -6, +6
Selected by minimizing per-group quantization error across six LLMs (Fig. 3). These values define the extended FP3 data types and are then used in the main perplexity evaluations on the same models.
FP4 special values = -5, +5, -8, +8
Selected by minimizing quantization error across the same six LLMs (Section III-A). Used in FP4-ER and FP4-EA for all 4-bit results.
Per-group scaling factor precision = INT8
Chosen because Table V shows INT8 reproduces FP16 perplexity on the benchmark models; lower precisions degrade accuracy. This empirically fitted choice is used in the hardware design and memory overhead analysis.
Group size = 128
Standard group size from prior work (AWQ, GPTQ); adopted rather than tuned, but affects quantization error and dequantization cycles.

assumptions (4)

domain assumption LLM weight tensors exhibit approximately Gaussian distributions with occasional asymmetric outliers at per-group granularity.
Motivates combining floating-point data types with asymmetric extension (Section II-C). Supported by prior studies and the authors' Fig. 2, but not derived.
domain assumption FP16 activation precision is sufficient during inference, and key and value tensors can be quantized to INT8 or INT4 with negligible accuracy loss.
The hardware maintains FP16 activations and quantizes key and value tensors (Section IV-B), relying on prior work [44], [52], [58].
domain assumption The unified bit-serial representation in Eq. (4) exactly represents all supported data types (INT8, INT6, FP4, FP3).
The decoder design assumes the conversion is exact; verified by RTL simulation, but the proof is not formal.
domain assumption CACTI and DRAMSim3 accurately model area, power, and DRAM energy for the evaluated accelerators.
Standard architecture evaluation tools; results depend on their accuracy.

how reviews work

0 comments

Cite this review

Pith. "Pith review of BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration." pith.science (2026). https://pith.science/paper/J3WSCGPM

@misc{pith2026241111745,
  author       = {Pith},
  title        = {Pith review of: BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J3WSCGPM}},
  note         = {Machine review of arXiv:2411.11745}
}

abstract

Large language models (LLMs) have demonstrated remarkable performance across various machine learning tasks. Yet the substantial memory footprint of LLMs significantly hinders their deployment. In this paper, we improve the accessibility of LLMs through BitMoD, an algorithm-hardware co-design solution that enables efficient LLM acceleration at low weight precision. On the algorithm side, BitMoD introduces fine-grained data type adaptation that uses a different numerical data type to quantize a group of (e.g., 128) weights. Through the careful design of these new data types, BitMoD is able to quantize LLM weights to very low precision (e.g., 4 bits and 3 bits) while maintaining high accuracy. On the hardware side, BitMoD employs a bit-serial processing element to easily support multiple numerical precisions and data types; our hardware design includes two key innovations: First, it employs a unified representation to process different weight data types, thus reducing the hardware cost. Second, it adopts a bit-serial dequantization unit to rescale the per-group partial sum with minimal hardware overhead. Our evaluation on six representative LLMs demonstrates that BitMoD significantly outperforms state-of-the-art LLM quantization and acceleration methods. For discriminative tasks, BitMoD can quantize LLM weights to 4-bit with $<\!0.5\%$ accuracy loss on average. For generative tasks, BitMoD is able to quantize LLM weights to 3-bit while achieving better perplexity than prior LLM quantization scheme. Combining the superior model performance with an efficient accelerator design, BitMoD achieves an average of $1.69\times$ and $1.48\times$ speedups compared to prior LLM accelerators ANT and OliVe, respectively.

Figures

Figures reproduced from arXiv: 2411.11745 by the authors.

**Figure 2.** Maximum value and value range for different quantization granularity. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Normalized weight quantization error (↓) with different special values (SV) for FP3. We use per-group quantization with a group size of 128. The special values ± 6 achieve the lowest overall quantization error, thus adopted in BitMoD. of low-precision data types. Hence, we limit the special value to low-precision integers. Furthermore, given N as the number of allowed special values, an encoding overhead of logN … view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Unified bit-serial representation of (a) INT8, INT6, and (b) FP4. Every [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: The microarchitecture of BitMoD PE. Every bit-serial weight term contains 1-bit sign (ws), 2-bit exponent (we), 1-bit mantissa (wm), and a shared bit-significance (wbsig). extensions in a unified architecture. However, the basic values of FP3 and FP4 use the floating-p…

**Figure 6.** Figure 6: BitMoD accelerator architecture. special values as needed, and the number of decoded bitserial terms can be minimized with simple modification to the decoder. For example, the special value 7 can be expressed as two bit-serial terms 2 3 and − 2 0 instead of using a le…

**Figure 7.** Figure 7: Speedup (↑) of different accelerators [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 8.** Figure 8: Energy consumption (↓) of different accelerators. “LL” and “LY” stand for ‘lossless” and ‘lossy”, respectively. Discriminative Tasks. Table VII compares the model accuracy of discriminative tasks when employing BitMoD and the baseline asymmetric integer quantization a…

**Figure 10.** Figure 10: Normalized area and power of BitMoD and different bit-parallel PEs. weights, which is an overkill for LLMs since the simple INT6 data type can achieve comparable accuracy under per-group weight quantization. Overall, the lossless BitMoD achieves 2.31× better energy ef…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

ITERA-LLM: Boosting Sub-8-Bit Large Language Model Inference via Iterative Tensor Decomposition
cs.AR 2025-05 conditional novelty 6.0 of 10

ITERA-LLM shows that iteratively decomposing quantized LLM weight matrices into low-rank factors and allocating ranks by BLEU sensitivity yields better accuracy-latency trade-offs than quantization alone on FPGAs.

Reference graph

Works this paper leans on

78 extracted references · 55 canonical work pages · cited by 1 Pith paper

[1]

01-ai/yi-6b

01-AI, “01-ai/yi-6b.” [Online]. Available: https://huggingface.co/01- ai/Yi-6B

work page
[2]

BitMoD Artifacts,

Abdelfattah Lab, “BitMoD Artifacts,” Nov. 2024. [Online]. Available: https://doi.org/10.5281/zenodo.14252531

work page doi:10.5281/zenodo.14252531 2024
[3]

Bit-pragmatic deep neural network computing,

J. Albericio, A. Delmas, P. Judd, S. Sharify, G. O’Leary, R. Genov, and A. Moshovos, “Bit-pragmatic deep neural network computing,” IEEE/ACM 50th Annual International Symposium on Microarchitecture (MICRO), 2017

work page 2017
[4]

QuaRot: Outlier-free 4-bit inference in rotated llms,

S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman, “QuaRot: Outlier-free 4-bit inference in rotated llms,” arXiv preprint arXiv:2404.00456 , 2024

arXiv 2024
[5]

FPRaker: A process- ing element for accelerating neural network training,

O. M. Awad, M. Mahmoud, I. E. Vivancos, A. H. Zadeh, C. Bannon, A. Jayarajan, G. Pekhimenko, and A. Moshovos, “FPRaker: A process- ing element for accelerating neural network training,” IEEE/ACM 54th Annual International Symposium on Microarchitecture (MICRO) , 2020

work page 2020
[6]

CACTI 7: New tools for interconnect exploration in innovative off-chip memories,

R. Balasubramonian, A. B. Kahng, N. Muralimanohar, A. Shafiee, and V . Srinivas, “CACTI 7: New tools for interconnect exploration in innovative off-chip memories,”ACM Trans. Archit. Code Optim., vol. 14, no. 2, June 2017

work page 2017
[7]

PIQA: Reason- ing about physical commonsense in natural language,

Y . Bisk, R. Zellers, R. L. Bras, J. Gao, and Y . Choi, “PIQA: Reason- ing about physical commonsense in natural language,” arXiv preprint arXiv:1911.11641, 2019

arXiv 1911
[8]

A signed binary multiplication technique,

A. D. Booth, “A signed binary multiplication technique,” Quarterly Journal of Mechanics and Applied Mathematics , vol. 4, pp. 236–240, 1951

work page 1951

Show all 78 references

[9]

Language Models are Few-Shot Learners,

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. C...

2020
[10]

QuIP: 2-bit quan- tization of large language models with guarantees,

J. Chee, Y . Cai, V . Kuleshov, and C. D. Sa, “QuIP: 2-bit quan- tization of large language models with guarantees,” arXiv preprint arXiv:2307.13304, 2023

2023 arXiv
[11]

EfficientQAT: Efficient quantization-aware training for large language models,

M. Chen, W. Shao, P. Xu, J. Wang, P. Gao, K.-C. Zhang, Y . Qiao, and P. Luo, “EfficientQAT: Efficient quantization-aware training for large language models,” arXiv preprint arXiv:2407.11062 , 2024

2024 arXiv
[12]

BBS: Bi-directional bit-level sparsity for deep learning acceleration,

Y . Chen, J. Meng, J.-S. Seo, and M. S. Abdelfattah, “BBS: Bi-directional bit-level sparsity for deep learning acceleration,” 57th IEEE/ACM Inter- national Symposium on Microarchitecture (MICRO) , 2024

2024
[13]

Think you have solved question answering? try ARC, the ai2 reasoning challenge,

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try ARC, the ai2 reasoning challenge,” arXiv preprint arXiv:1803.05457 , 2018

2018 arXiv
[14]

VS-Quant: Per-vector scaled quantization for accurate low-precision neural network inference,

S. Dai, R. Venkatesan, H. Ren, B. Zimmer, W. J. Dally, and B. Khailany, “VS-Quant: Per-vector scaled quantization for accurate low-precision neural network inference,” in Proceedings of Machine Learning and Systems (MLSys), 2021

2021
[15]

LLM.int8(): 8-bit matrix multiplication for transformers at scale,

T. Dettmers, M. Lewis, Y . Belkada, and L. Zettlemoyer, “LLM.int8(): 8-bit matrix multiplication for transformers at scale,” arXiv preprint arXiv:2208.07339, 2022

2022 arXiv
[16]

8-bit optimizers via block-wise quantization,

T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer, “8-bit optimizers via block-wise quantization,” arXiv preprint arXiv:2110.02861 , 2022

2022 arXiv
[17]

QLoRA: Efficient finetuning of quantized llms,

T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient finetuning of quantized llms,” arXiv:2305.14314, 2023

2023 arXiv
[18]

Documenting large webtext corpora: A case study on the colossal clean crawled corpus,

J. Dodge, A. Marasovic, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner, “Documenting large webtext corpora: A case study on the colossal clean crawled corpus,” in Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021

2021
[19]

Learning from students: Applying t-distributions to explore accurate and efficient formats for llms,

J. Dotzel, Y . Chen, B. Kotb, S. Prasad, G. Wu, S. Li, M. S. Abdelfat- tah, and Z. Zhang, “Learning from students: Applying t-distributions to explore accurate and efficient formats for llms,” arXiv preprint arXiv:2405.03103, 2024

2024 arXiv
[20]

GPTQ: Accurate post-training compression for generative pretrained transformers,

E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate post-training compression for generative pretrained transformers,” arXiv preprint arXiv:2210.17323, 2022

2022 arXiv
[21]

A framework for few-shot language model evaluation,

L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, “A framework...
[22]

Ai and memory wall,

A. Gholami, Z. Yao, S. Kim, C. Hooper, M. W. Mahoney, and K. Keutzer, “Ai and memory wall,” IEEE Micro, 2024

2024
[23]

SparTen: A sparse tensor accelerator for convolutional neural net- works,

A. Gondimalla, N. Chesnut, M. Thottethodi, and T. N. Vijaykumar, “SparTen: A sparse tensor accelerator for convolutional neural net- works,” Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) , 2019

2019
[24]

Eureka: Efficient tensor cores for one-sided unstructured sparsity in dnn inference,

A. Gondimalla, M. Thottethodi, and T. N. Vijaykumar, “Eureka: Efficient tensor cores for one-sided unstructured sparsity in dnn inference,” 56th IEEE/ACM International Symposium on Microarchitecture (MICRO) , 2023

2023
[25]

OliVe: Accelerating large language models via hardware-friendly outlier-victim pair quantization,

C. Guo, J. Tang, W. Hu, J. Leng, C. Zhang, F. Yang, Y .-B. Liu, M. Guo, and Y . Zhu, “OliVe: Accelerating large language models via hardware-friendly outlier-victim pair quantization,” ACM/IEEE 50th Annual International Symposium on Computer Architecture (ISCA) , 2023

2023
[26]

ANT: Exploiting adaptive numerical data type for low-bit deep neural network quantization,

C. Guo, C. Zhang, J. Leng, Z. Liu, F. Yang, Y .-B. Liu, M. Guo, and Y . Zhu, “ANT: Exploiting adaptive numerical data type for low-bit deep neural network quantization,” IEEE/ACM 55th Annual International Symposium on Microarchitecture (MICRO) , 2022

2022
[27]

FIGNA: Integer unit-based accelerator design for fp-int gemm preserving numerical accuracy,

J. Jang, Y . Kim, J. Lee, and J.-J. Kim, “FIGNA: Integer unit-based accelerator design for fp-int gemm preserving numerical accuracy,”IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2024

2024
[28]

Stripes: Bit-serial deep neural network computing,

P. Judd, J. Albericio, and A. Moshovos, “Stripes: Bit-serial deep neural network computing,” 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2016

2016
[29]

DRAMsim3: A cycle-accurate, thermal-capable dram simulator,

S.-J. Li, Z. Yang, D. Reddy, A. Srivastava, and B. Jacob, “DRAMsim3: A cycle-accurate, thermal-capable dram simulator,” IEEE Computer Architecture Letters, vol. 19, pp. 106–109, 2020

2020
[30]

AWQ: Activation-aware weight quanti- zation for llm compression and acceleration,

J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “AWQ: Activation-aware weight quanti- zation for llm compression and acceleration,” in Proceedings of Machine Learning and Systems (MLSys) , 2024

2024
[31]

LLM-QAT: Data-free quan- tization aware training for large language models,

Z. Liu, B. O ˘guz, C. Zhao, E. Chang, P. Stock, Y . Mehdad, Y . Shi, R. Krishnamoorthi, and V . Chandra, “LLM-QAT: Data-free quan- tization aware training for large language models,” arXiv preprint arXiv:2305.17888, 2023

2023 arXiv
[32]

Torch2Chip: An end-to-end customizable deep neural network compression and deployment toolkit for prototype hardware accelerator design,

J. Meng, Y . Liao, A. Anupreetham, A. Hasssan, S. Yu, H.-s. Suh, X. Hu, and J.-s. Seo, “Torch2Chip: An end-to-end customizable deep neural network compression and deployment toolkit for prototype hardware accelerator design,” in Proceedings of Machine Learning and Systems (MLS...

2024
[33]

Pointer sentinel mixture models,

S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843 , 2016

2016 arXiv
[34]

Meta llama

Meta, “Meta llama.” [Online]. Available: https://github.com/meta- llama/llama
[35]

Meta llama 3

Meta, “Meta llama 3.” [Online]. Available: https://github.com/meta- llama/llama3
[36]

microsoft/phi-2

Microsoft, “microsoft/phi-2.” [Online]. Available: https://huggingface. co/microsoft/phi-2
[37]

Jetson TX2 Module

NVIDIA, “Jetson TX2 Module.” [Online]. Available: https://developer. nvidia.com/embedded/jetson-tx2
[38]

OCP Microscaling Formats (MX) Specification

Open Compute Project, “OCP Microscaling Formats (MX) Specification.” [Online]. Available: https://www.opencompute.org/ documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf
[39]

Energy-efficient neural network accel- erator based on outlier-aware low-precision computation,

E. Park, D. Kim, and S. Yoo, “Energy-efficient neural network accel- erator based on outlier-aware low-precision computation,” ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), 2018

2018
[40]

With shared microexponents, a little shifting goes a long way,

B. D. Rouhani, R. Zhao, V . Elango, R. Shafipour, M. Hall, M. Mes- makhosroshahi, A. More, L. Melnick, M. Golub, G. Varatkar, L. Shao, G. Kolhe, D. Melts, J. Klar, R. L’Heureux, M. Perry, D. Burger, E. S. Chung, Z. Deng, S. Naghshineh, J. Park, and M. Naumov, “With shared micr...

2023
[41]

WinoGrande: An adversarial winograd schema challenge at scale,

K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “WinoGrande: An adversarial winograd schema challenge at scale,” arXiv preprint arXiv:1907.10641, 2019

1907 arXiv
[42]

OmniQuant: Omnidirectionally calibrated quan- tization for large language models,

W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y . J. Qiao, and P. Luo, “OmniQuant: Omnidirectionally calibrated quan- tization for large language models,” arXiv preprint arXiv:2308.13137 , 2024

2024 arXiv
[43]

Laconic deep learning inference acceleration,

S. Sharify, A. D. Lascorz, M. Mahmoud, M. Nikolic, K. Siu, D. M. Stuart, Z. Poulos, and A. Moshovos, “Laconic deep learning inference acceleration,” ACM/IEEE 46th Annual International Symposium on Computer Architecture (ISCA), 2019

2019
[44]

FlexGen: High-throughput generative inference of large language models with a single gpu,

Y . Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, D. Y . Fu, Z. Xie, B. Chen, C. W. Barrett, J. Gonzalez, P. Liang, C. R ´e, I. Stoica, and C. Zhang, “FlexGen: High-throughput generative inference of large language models with a single gpu,” in International Conference on Mach...

2023
[45]

BitWave: Exploiting column-based bit-level sparsity for deep learning accelera- tion,

M. Shi, V . Jain, A. Joseph, M. Meijer, and M. Verhelst, “BitWave: Exploiting column-based bit-level sparsity for deep learning accelera- tion,” Proceedings of the 30th IEEE International Symposium on High- Performance Computer Architecture (HPCA) , 2024

2024
[46]

Recursive deep models for semantic compositionality over a sentiment treebank,

R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2013

2013
[47]

Llama: Open and efficient foundation language models,

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

2023 arXiv
[48]

Dual-side sparse tensor core,

Y . Wang, C. Zhang, Z. Xie, C. Guo, Y . Liu, and J. Leng, “Dual-side sparse tensor core,” ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA) , 2021

2021
[49]

ZeroQuant(4+2): Redefining llms quantization with a new fp6-centric strategy for diverse generative tasks,

X. Wu, H. Xia, S. Youn, Z. Zheng, S. Chen, A. Bakhtiari, M. Wyatt, R. Y . Aminabadi, Y . He, O. Ruwase, L. Song, and Z. Yao, “ZeroQuant(4+2): Redefining llms quantization with a new fp6-centric strategy for diverse generative tasks,” arXiv preprint arXiv:2312.08583 , 2023

2023 arXiv
[50]

HighLight: Efficient and flexible dnn acceleration with hierar- chical structured sparsity,

Y . N. Wu, P.-A. Tsai, S. Muralidharan, A. Parashar, V . Sze, and J. S. Emer, “HighLight: Efficient and flexible dnn acceleration with hierar- chical structured sparsity,” 56th IEEE/ACM International Symposium on Microarchitecture (MICRO), 2023

2023
[51]

Quant-llm: Accelerating the serving of large language models via fp6- centric algorithm-system co-design on modern gpus,

H. Xia, Z. Zheng, X. Wu, S. Chen, Z. Yao, S. Youn, A. Bakhtiari, M. Wyatt, D. Zhuang, Z. Zhou, O. Ruwase, Y . He, and S. L. Song, “Quant-llm: Accelerating the serving of large language models via fp6- centric algorithm-system co-design on modern gpus,” inUSENIX Annual Technica...

2024
[52]

SmoothQuant: Accurate and efficient post-training quantization for large language models,

G. Xiao, J. Lin, M. Seznec, J. Demouth, and S. Han, “SmoothQuant: Accurate and efficient post-training quantization for large language models,” arXiv preprint arXiv:2211.10438 , 2022

2022 arXiv
[53]

ZeroQuant: Efficient and affordable post-training quantization for large- scale transformers,

Z. Yao, R. Y . Aminabadi, M. Zhang, X. Wu, C. Li, and Y . He, “ZeroQuant: Efficient and affordable post-training quantization for large- scale transformers,” arXiv preprint arXiv:2206.01861 , 2022

2022 arXiv
[54]

GOBO: Quantizing attention-based nlp models for low latency and energy efficient inference,

A. H. Zadeh, I. Edo, O. M. Awad, and A. Moshovos, “GOBO: Quantizing attention-based nlp models for low latency and energy efficient inference,” IEEE/ACM 53rd Annual International Symposium on Microarchitecture (MICRO), 2020

2020
[55]

Mokey: enabling narrow fixed-point inference for out-of-the-box floating-point transformer models,

A. H. Zadeh, M. Mahmoud, A. Abdelhadi, and A. Moshovos, “Mokey: enabling narrow fixed-point inference for out-of-the-box floating-point transformer models,” ACM/IEEE 49th Annual International Symposium on Computer Architecture (ISCA) , 2022

2022
[56]

HellaSwag: Can a machine really finish your sentence?

R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “HellaSwag: Can a machine really finish your sentence?” in Annual Meeting of the Association for Computational Linguistics (ACL) , 2019

2019
[57]

Opt: Open pre-trained transformer language models,

S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. T. Diab, X. Li, X. V . Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “Opt: Open pre-trained transformer language models,” arXiv prepri...

2022 arXiv
[58]

Atom: Low-bit quantization for efficient and accurate llm serving,

Y . Zhao, C.-Y . Lin, K. Zhu, Z. Ye, L. Chen, S. Zheng, L. Ceze, A. Kr- ishnamurthy, T. Chen, and B. Kasikci, “Atom: Low-bit quantization for efficient and accurate llm serving,” in Proceedings of Machine Learning and Systems (MLSys) , 2024

2024
[59]

Cambricon-S: Addressing irregularity in sparse neural networks through a cooperative software/hardware approach,

X. Zhou, Z. Du, Q. Guo, S. Liu, C. Liu, C. Wang, X. Zhou, L. Li, T. Chen, and Y . Chen, “Cambricon-S: Addressing irregularity in sparse neural networks through a cooperative software/hardware approach,” 51st Annual IEEE/ACM International Symposium on Microarchitecture (MICRO),...

2018
[61]

How to Access: We maintain a publicly-available repos- itory [2] where we have open-sourced all of our artifacts
[62]

Hardware Dependencies: One NVIDIA GPU with at least 40 GB of VRAM (e.g., A6000), in addition to a normal desktop computer with at least 512 GB of free disk space
[63]

The quantization experiments require CUDA

Software Dependencies: All experiments require Conda for managing virtual Python environments. The quantization experiments require CUDA. Other requirements are automat- ically installed by scripts in the following sections. When running experiments, please use a tmux session ...
[64]

This can reproduce the results in Table VI and Table VIII

bitmod-quant, which runs the baseline weight-only quantization with different data types. This can reproduce the results in Table VI and Table VIII
[65]

This can reproduce the results in Fig

bitmod-sim, contains a custom simulator that calculates the latency and energy of the BitMoD accelerator. This can reproduce the results in Fig. 7 and Fig. 8
[66]

This can reproduce the AWQ results in Table XI

A WQ-BitMoD, which runs AWQ [30] with integer and BitMoD data types. This can reproduce the AWQ results in Table XI
[67]

This can reproduce the OmniQuant results in Table XI

OmniQuant-BitMoD, which runs OmniQuant [42] with integer and BitMoD data types. This can reproduce the OmniQuant results in Table XI
[68]

This can reproduce the results in Table XII

SmoothQuant-BitMoD, which runs SmoothQuant [52] with integer and BitMoD data types for weight quantiza- tion. This can reproduce the results in Table XII. Please go to every sub-folder and refer to the corresponding ‘README.md’ for detailed setup instructions. Note that AWQ, O...
[69]

$ cd bitmod quant $ conda activate awq−bitmod In ‘ run_exp.sh’, modify the ‘ export’ command by specifying the HuggingFace home directory, ‘ HF_HOME’, on your computer

BitMoD Weight-only Quantization: Run the basic LLM weight-only quantization experiments to reproduce the results in Table VI and Table VIII. $ cd bitmod quant $ conda activate awq−bitmod In ‘ run_exp.sh’, modify the ‘ export’ command by specifying the HuggingFace home director...
[70]

When enabled / disabled, it will evaluate the hardware performance of generative / discriminative tasks

BitMoD Hardware Simulation: Before running the sim- ulator, go to ‘ bitmod_sim’ of the repository: $ cd bitmod sim $ conda activate awq−bitmod In ‘ run_shape_profile.sh’, modify the ‘ export’ command by specifying the HuggingFace home directory, ‘HF_HOME’, on your computer: $ ...
[71]

You can compare these with the AWQ results in Table XI

AWQ: Go to the ‘ AWQ-BitMoD’ directory: $ cd AWQ−BitMoD $ conda activate awq−bitmod In ‘ run_awq.sh’ and ‘ run_eval_ppl.sh’, modify the first ‘ export’ command by specifying the HuggingFace home directory, ‘ HF_HOME’, on your computer: $ export HF HOME=”your/HF HOME/directory”...
[72]

OmniQuant: Go to the ‘ OmniQuant-BitMoD’ direc- tory: $ cd OmniQuant−BitMoD $ conda activate omniquant−bitmod The comprehensive scripts to reproduce the Table XI Om- niQuant results are available in the ‘ scripts’ directory. Before running any command in the scripts, execute t...
[73]

You can compare these results with the SmoothQuant results in Table XII

SmoothQuant: Go to the ‘ SmoothQuant-BitMoD’ directory: $ cd SmoothQuant−BitMoD $ conda activate smoothquant−bitmod In ‘ run_experiments.sh’, modify the ‘ export’ command by specifying the HuggingFace home directory, ‘HF_HOME’, on your computer: $ export HF HOME=”your/HF HOME/...
[74]

bitmod-quant/results_quant, contains the per- plexity results in Table VI and Table VIII
[75]

7 and Fig

bitmod-sim/plot, contains two Jupyter notebooks to reproduce Fig. 7 and Fig. 8, respectively
[76]

AWQ-BitMoD/results, contains the AWQ results in Table XI
[77]

OmniQuant-BitMoD/log, contains the OmniQuant results in Table XI
[78]

SmoothQuant-BitMoD/results_mod, contains the SmoothQuant results in Table XII. G. Methodology Submission, reviewing and badging methodology: • https://www.acm.org/publications/policies/artifact- review-and-badging-current • https://cTuning.org/ae
[2023]

Available: https://zenodo.org/records/10256836

[Online]. Available: https://zenodo.org/records/10256836

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

01-ai/yi-6b

01-AI, “01-ai/yi-6b.” [Online]. Available: https://huggingface.co/01- ai/Yi-6B

work page

[2] [2]

BitMoD Artifacts,

Abdelfattah Lab, “BitMoD Artifacts,” Nov. 2024. [Online]. Available: https://doi.org/10.5281/zenodo.14252531

work page doi:10.5281/zenodo.14252531 2024

[3] [3]

Bit-pragmatic deep neural network computing,

J. Albericio, A. Delmas, P. Judd, S. Sharify, G. O’Leary, R. Genov, and A. Moshovos, “Bit-pragmatic deep neural network computing,” IEEE/ACM 50th Annual International Symposium on Microarchitecture (MICRO), 2017

work page 2017

[4] [4]

QuaRot: Outlier-free 4-bit inference in rotated llms,

S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman, “QuaRot: Outlier-free 4-bit inference in rotated llms,” arXiv preprint arXiv:2404.00456 , 2024

arXiv 2024

[5] [5]

FPRaker: A process- ing element for accelerating neural network training,

O. M. Awad, M. Mahmoud, I. E. Vivancos, A. H. Zadeh, C. Bannon, A. Jayarajan, G. Pekhimenko, and A. Moshovos, “FPRaker: A process- ing element for accelerating neural network training,” IEEE/ACM 54th Annual International Symposium on Microarchitecture (MICRO) , 2020

work page 2020

[6] [6]

CACTI 7: New tools for interconnect exploration in innovative off-chip memories,

R. Balasubramonian, A. B. Kahng, N. Muralimanohar, A. Shafiee, and V . Srinivas, “CACTI 7: New tools for interconnect exploration in innovative off-chip memories,”ACM Trans. Archit. Code Optim., vol. 14, no. 2, June 2017

work page 2017

[7] [7]

PIQA: Reason- ing about physical commonsense in natural language,

Y . Bisk, R. Zellers, R. L. Bras, J. Gao, and Y . Choi, “PIQA: Reason- ing about physical commonsense in natural language,” arXiv preprint arXiv:1911.11641, 2019

arXiv 1911

[8] [8]

A signed binary multiplication technique,

A. D. Booth, “A signed binary multiplication technique,” Quarterly Journal of Mechanics and Applied Mathematics , vol. 4, pp. 236–240, 1951

work page 1951