Pith. sign in

REVIEW 3 major objections 5 minor 61 references

OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OASIS claims non-uniform 4-bit weights and activations can be multiplied directly via precomputed Cartesian-product LUTs, eliminating dequantization, with ~2% accuracy loss and 3.00x speedup.

desk verdict OASIS has a genuinely new LUT-based GEMM idea for non-uniform W4A4, but a load-bearing gap in how per-channel/per-token scales are handled makes the core hardware claim unverifiable as written. read the letter →

arxiv 2507.23035 v4 pith:PRM6IJCO submitted 2025-07-30 cs.LG cs.AR

Xueying Wu , Baijun Zhou , Zhihui Gao , Yuzhe Fu , Qilin Zheng , Yintao He , Hai Li This is my paper
classification cs.LGcs.AR
keywords largelanguagemodelsnon-uniformquantizationlearnedcodebooklookuptableGEMMW4A4activationoutliershardwareaccelerationlow-bitinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that non-uniform weight-and-activation quantization, the kind where 4-bit indices point into learned lists of representable values, does not have to cost accuracy or require dequantizing back to high precision before every matrix multiplication. The proposed scheme, OASIS, stores all products of the weight and activation codebook values in a lookup table, so a general matrix multiplication (GEMM) between quantized operands becomes a count-and-weight operation, with a parallel branch correcting the few activation outliers that would otherwise cause large errors. If the paper is right, the standard trade-off between integer weight-and-activation quantization, which is fast but lossy, and non-uniform quantization, which is accurate but slow, is broken, making 4-bit and even 3-bit activation inference practical. The paper reports an average accuracy drop of about 2% versus FP16 at W4A4, and, in simulation, a 3.00x throughput gain and 1.44x energy efficiency gain over a prior LUT-based accelerator.

What carries the argument

The load-bearing object is the Cartesian Product LUT: a precomputed table of $2^{n_W+n_A}$ values, one for every pair of weight-centroid and activation-centroid. Its size depends only on bit widths, not on GEMM reduction length $K$, so an entire $K$-length inner product can be replaced by a histogram of concatenated indices and a weighted sum of LUT entries; this is what makes the claimed 64x LUT-size reduction and 1024x parallelism increase possible. The second mechanism is look-ahead outlier compensation, which splits computation into a main branch that quantizes every activation and a parallel branch that fetches the top and bottom 0.5% activation outliers, computes their residuals in FP16, multiplies each residual by the corresponding dequantized weight row, and adds the correction, hiding detection latency. The third is Orizuru, a top-$k$ detection engine built from two complete binary trees with shared leaves that extracts the $k$ largest and $k$ smallest elements of an $N$-value activation token in $1.5N + 2k \log_2 N$ comparisons, several times fewer than the comparison count of a prior top-$k$ engine.

What would settle it

Run the W4A4 pipeline as specified, with 16 C4 calibration samples and the top and bottom 0.5% of activations kept in FP16, on a held-out corpus from a domain far from C4, such as code or biomedical text, and compare its accuracy to the FP16 baseline; if the gap systematically exceeds the reported ~2% average, the offline-centroid transferability assumption is broken. On the hardware side, an implementation that cannot reproduce the paper's simulated cycle counts for a 1-4096-4096 GEMM at 500 MHz would falsify the claimed 3.00x speedup over FIGLUT.

Watch

Extended reading notes

Core claim

On its own terms, OASIS claims to execute the GEMM between learned-codebook-quantized weights and activations with no dequantization anywhere in the main path. During offline preparation it builds a Cartesian Product LUT containing every product $w_i a_j$ of a weight centroid and an activation centroid; with $n_W = n_A = 4$ this is only $2^{8} = 256$ entries. At runtime, the accelerator concatenates the 4-bit weight index and 4-bit activation index for each element, counts the occurrences of each concatenated index along the reduction dimension $K$, and forms the output as a weighted sum of the corresponding LUT entries, so the number of additions per output channel falls from $K$ to $2^{n_W+n_A}$. Because both codebooks are learned offline, the LUT is independent of both streaming activations and reduction length, which removes on-the-fly LUT generation and permits much larger compute granularity than prior weight-only LUT GEMMs. To keep accuracy under aggressive activation quantization, OASIS marks the top and bottom 0.5% of each token's activations as outliers, preserves them in FP16, and runs a concurrent error-compensation branch whose residual products with the dequantized weight rows are added to the main branch; the paper states this yields mathematically identical results to a conventional high-precision outlier path without putting outlier detection on the critical path. Across OPT, LLaMA, LLaMA-2/3, and Mistral models, the reported W4A4 accuracy drop relative to FP16 is around 1.94-2.05% on average.

Load-bearing premise

The accuracy result rests on activation centroids learned offline from only 16 C4 samples continuing to match the activations encountered at inference time, so the reported ~2% drop may not hold if real-world input distributions shift.

Editorial extensions

If this is right

  • W4A4 and W4A3 learned-codebook quantization become directly executable, so LLM inference can keep non-uniform quantization's accuracy without paying the FP16 dequantization cost that dominates weight-only quantization.
  • Dynamic outlier detection, which prior work found more accurate than static outlier channels, can be used without lengthening the GEMM critical path.
  • Because the Cartesian-product LUT size does not grow with model width, the throughput advantage should become larger on very wide GEMMs, such as those in 70B-class models, rather than evaporating at scale.
  • In the low-batch decode regime where edge LLM serving operates, the architecture is claimed to sustain 3.00x throughput and 1.44x energy efficiency over the FIGLUT accelerator in cycle-accurate simulation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension, not made by the paper, is to apply the same Cartesian-product LUT trick to other learned-codebook tensors, such as KV caches, embedding tables, or intermediate activations, wherever both the codebook and the value distribution are known ahead of time; each would become a precomputable table rather than a dequantization path.
  • The reported accuracy gap between OASIS-S, which reuses offline outlier thresholds, and OASIS, which detects outliers dynamically, suggests dynamic detection will matter more on inputs that drift from the calibration corpus; a testable prediction is that OASIS-S degrades faster than OASIS on out-of-distribution text.
  • If Orizuru's comparison count holds at scale, top-$k$ outlier selection is cheap enough to become a standard per-token hardware primitive that could benefit quantization schemes beyond this specific accelerator; the paper does not claim this generalization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. OASIS proposes a lookup-table (LUT) based GEMM scheme for non-uniformly quantized LLM inference, combining a precomputed Cartesian-product LUT of weight and activation centroids with a parallel outlier-compensation branch. The main branch avoids dequantization by counting concatenated weight/activation index pairs and reducing to a weighted sum of LUT entries; the outlier branch detects top-k activation outliers and adds their residuals in FP16. The paper also contributes Orizuru, a tree-based top-k detector, and an accelerator architecture with 16 PE lines, index counters, clustering units, and a 256-entry LUT. Evaluations report roughly 2% average zero-shot accuracy drop versus FP16 and 3.00x throughput / 1.44x energy efficiency over the FIGLUT accelerator in simulation.

Significance. If the described computation is correct, this is a significant algorithm-architecture co-design result. The core mathematical decomposition—look-ahead GEMM on quantized activations plus residual error compensation for outliers—is exact by construction, and the claimed 256-entry Cartesian-product LUT is a genuine improvement over the group-size-dependent inner-product LUTs of prior WOQ LUT-GEMM designs. The LUT-size and FLOP arithmetic in Table I and Fig. 16 is internally consistent, and the evaluation is broad, covering 11 model configurations, perplexity, and six zero-shot tasks. The Orizuru detector's comparison count of 1.5N + 2k log2(N) is low and the hardware study uses cycle-accurate simulation with 28 nm synthesis-based area/power numbers. These strengths make the work worth serious consideration, provided the central scaling-factor pipeline gap and the inconsistent accuracy reporting are resolved.

major comments (3)
  1. [Section III-A, Section III-B, Fig. 6, Fig. 8] Section III-A specifies per-output-channel weight scaling factors and per-token activation scaling factors, but Section III-B and the hardware pipeline in Fig. 8 compute the LUT-GEMM result as a weighted sum of raw Cartesian products of shared centroids, with no step that applies those scale factors. For a token t and output channel o, the true quantized product is s_a(t) * s_w(o) * C_w[idx_w] * C_a[idx_a]; the described LUT-GEMM computes only C_w * C_a, so the outputs are missing the scale product. If the scales are meant to be applied after the LUT reduction, that step is never described in the five-step pipeline of Fig. 8 or in the example of Fig. 6. If instead the LUT is meant to store scaled products, it would have to be replicated per output channel and, because s_a(t) varies per token, would depend on runtime activations, contradicting the claimed offline precomputation and 256-entry LUT size. This is a load-bearing gap in the description of the core computation and must be clarified or corrected.
  2. [Abstract, Section I, Section V-B, Table IV] The reported accuracy improvement is inconsistent across the front matter and the evaluation. The abstract states an average accuracy drop of 1.98% and a 5.18% lower drop than Atom; the Introduction states 1.94% and 6.34%; Section V-B states a 2.05% average drop at W4A4 and 5.90% at W4A3; and the zero-shot averages in Table IV for LLaMA-2-7B, LLaMA-3-8B, and Mistral at W4A4 give (1.94 + 3.03 + 1.19)/3 = 2.05%. Please reconcile these numbers and state precisely the metric and the model set over which the headline accuracy drop is computed.
  3. [Section V-C, Fig. 11] The hardware speedup claim against FIGLUT is not precision-matched: Fig. 11 compares OASIS-A4/A3 (W4A4/W4A3) against FIGLUT evaluated at W4A16. A substantial portion of the reported 3.00x throughput and 1.44x energy improvement may therefore be due to the reduced activation precision rather than to the LUT-based architecture itself. Please provide a W4A4-capable FIGLUT baseline, or an ablation that separates the precision contribution from the architectural contribution, so that the 'over FIGLUT' claim is not overstated.
minor comments (5)
  1. [Section III-A, Fig. 3, Table III] Fig. 3 shows low similarity between online and offline outlier thresholds (RMSE 0.32-0.38), yet the static variant OASIS-S in Table III performs only slightly worse than dynamic OASIS. Please explain this apparent tension, since it bears on the value of the dynamic detector.
  2. [Section V-A, Table II] The LUT is reported as 2 KB per chip, but a 256-entry FP16 Cartesian-product table occupies 512 bytes; please clarify what additional content (e.g., codebooks, tags) is stored in the LUT.
  3. [Section IV-D] The Orizuru comparison count is stated as 1.5N + 2k log2(N), but the max-tree initialization requires N-1 comparisons and the min tree reuses the last level, giving 1.5N-2 comparisons before popping; the stated formula slightly overcounts.
  4. [Table I] The LUT-size formula in Table I is hard to parse in the current formatting; please typeset it unambiguously to make the 64x reduction argument clear.
  5. [Section II-B] The '1024x higher computational parallelism' claim is configuration-specific (K=4096 versus group size 4); please state this dependence explicitly wherever the number is quoted.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the LUT construction is explicit, the evaluation is empirical against external baselines, and the only self-citations are background. The scaling-factor omission is a correctness gap, not a circular step.

full rationale

The paper's central claims are the 64x LUT-size reduction, dequantization-free GEMM, and the outlier look-ahead/error-compensation scheme. These are supported by an explicit construction (Cartesian Product LUT of shared centroids, Section III-B) and by empirical evaluation against external baselines (Atom, QuaRot, FP16, FIGLUT). No derivation reduces to its own inputs: the activation centroids are fitted on C4 and then evaluated on WikiText-2/zero-shot tasks, which is external validation, not circularity; the LUT is built from the learned codebooks, and the GEMM result is read off the LUT, not regressed to a target. The offline/online centroid-consistency test (Fig. 5) is empirical and uses different datasets for fitting and checking. The Orizuru comparison counts are derived from a standard tournament-tree analysis, not from a fitted parameter. The paper does contain minor self-citations ([57], and possibly [14]) used only for background claims about WAQ accuracy degradation; these are not load-bearing and do not constitute circular evidence. One non-circular concern deserves flagging: Section III-A defines per-output-channel weight scales and per-token activation scales, while Section III-B constructs a single Cartesian Product LUT of unscaled centroid products, and the hardware pipeline (Fig. 8 steps 1-5) contains no scaling operation. The LUT-GEMM output is therefore missing the s_w[out_channel]*s_a[token] factors unless those scales are silently folded elsewhere; this is an omitted proof/internal-inconsistency in the description, not a circular step, because the LUT output is not asserted to equal the scaled product by definition—it is simply underspecified. Overall, no significant circularity; score 2 reflects the minor self-citations and the unresolved scaling description.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

The performance claims rest on three assumptions: (1) offline-learned activation centroids generalize to online data, (2) the cycle-accurate simulator faithfully models OASIS and baselines, and (3) a 1% FP16 outlier budget is sufficient. The LUT decomposition itself is exact and needs no fitting. The activation codebook values and the chosen outlier percentage are calibrated to the evaluation datasets.

free parameters (3)
  • Activation outlier percentage = 0.5% top and 0.5% bottom (1% total)
    Hand-chosen threshold for treating activations as outliers; sensitivity analysis in Fig. 15 shows PPL improves with more outliers but throughput degrades beyond 1%. The claim of negligible runtime overhead is calibrated to this 1% operating point.
  • Calibration samples for activation centroids = 16 C4 samples
    Chosen via Fig. 17 as the knee of the PPL versus calibration cost curve. The accuracy results depend on these learned centroids.
  • PE Line resource allocation = 32 Index Counters, 8 MAC units per line, 16 PE lines
    Hardware counts chosen so main and outlier branch latencies balance at 1% outliers (stated in Section V-D4). These choices directly shape the reported 3.00x speedup over FIGLUT.
assumptions (5)
  • domain assumption Offline-learned activation centroids from 16 C4 samples generalize to online inference distributions.
    Invoked in Section III-A and validated only on WikiText-2 and six commonsense datasets with a single-layer comparison in Fig. 5. Distribution shift (code, chat, multilingual) is not tested.
  • domain assumption K-Means quantized weights without outlier protection retain sufficient accuracy at 4-bit.
    Assumed throughout Section III-A; no weight outlier protection is used, and the accuracy results support it for the tested models.
  • domain assumption The cycle-accurate simulator modified from DnnWeaver, with Cacti and DRAMSim3, faithfully models OASIS and FIGLUT performance.
    All hardware speedup and energy claims in Section V-C rely on this simulator; no RTL, FPGA prototype, or tape-out is provided.
  • standard math The Cartesian product LUT decomposition exactly matches quantized weight-activation products.
    By construction, the LUT stores all 2^(nW+nA) products of weight and activation centroids; the weighted sum of counts recovers the GEMM exactly.
  • domain assumption Representing outliers in FP16 with 1% budget is sufficient to preserve model accuracy.
    Validated empirically in Table III/IV; the choice depends on model and task, and the paper does not test higher outlier budgets in accuracy experiments.
invented entities (2)
  • Orizuru top-k detection engine
    purpose: Dynamically identify top-k largest and smallest activation values per token in real time.
    Only simulated at 28nm with synthesis results; no fabricated silicon or FPGA prototype. Its comparison count claims are analytical.
  • WAQ Cartesian Product LUT-GEMM scheme
    purpose: Performs GEMM between non-uniformly quantized weights and activations without dequantization.
    This is the central new computation scheme; its efficiency is validated only through the authors' simulator, not independent hardware measurement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration." pith.science (2026). https://pith.science/paper/PRM6IJCO

@misc{pith2026250723035,
  author       = {Pith},
  title        = {Pith review of: OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PRM6IJCO}},
  note         = {Machine review of arXiv:2507.23035}
}
read the original abstract

Large language models (LLMs) have demonstrated impressive capabilities across a wide range of applications, but demand substantial memory and compute resources during inference. Existing quantization methods expose a trade-off between efficiency and accuracy: weight-only quantization (WOQ) incurs costly dequantization overheads, while integer weight-and-activation quantization (INT-WAQ) reduces precision and degrades model quality. Non-uniform weight-and-activation quantization (NU-WAQ) can better capture the non-uniform distributions of LLM weights and activations, yet remains incompatible with conventional low-precision compute units. This paper presents OASIS, a lookup table (LUT)-based architecture that enables efficient general matrix multiplication (GEMM) between non-uniformly quantized weights and activations without requiring dequantization. OASIS employs pre-computed Cartesian Product LUTs, achieving a 64x reduction in LUT size and enabling a 1024x higher computational parallelism over existing LUT-based GEMM methods. To preserve accuracy under aggressive activation quantization, OASIS introduces an outlier-aware quantization scheme with concurrent LUT-based GEMM and error compensation for outliers. Furthermore, we design Orizuru, an efficient top-k detection engine for real-time activation outlier identification. According to extensive evaluations, OASIS incurs an average accuracy drop of only 1.98% compared to the FP16 baseline, which is 5.18% lower than Atom. On the hardware side, OASIS achieves an average 3.00x speedup and a 1.44x energy efficiency improvement compared to the FIGLUT accelerator.

Figures

Figures reproduced from arXiv: 2507.23035 by the authors.

Figure 1
Figure 1. Comparison of GEMM schemes: (a) WINT4AFP16 FP16 GEMM, (b) WINT4AINT4 INT4 GEMM, (c) WNU4ANU4 FP16 GEMM, (d) WNU4ANU4 LUT-GEMM (Ours). “NU” refers to non-uniform quantization. Our proposed design features two parallel computation branches that separately handle inlier and outlier operations. Our LUT-based GEMM design enables matrix multiplications between non-uniformly quantized weights and activations without requir… view at source ↗
Figure 2
Figure 2. (a) Existing WOQ LUT-GEMM scheme. A, B, C, and D denote different streaming activation values. (b) Our proposed WAQ LUT-GEMM scheme. ai and wi denote the activation and weight centroids, respectively. TABLE I COMPARISON OF LUT-BASED GEMM SCHEMES. WOQ LUT-GEMM Ours Wgt. Precision (nW ) NU4 NU4 Act. Precision (nA) FP16 NU4 Offline-Computed LUT? ✗ ✓ Group Size µ K LUT Size 2 µ · K µ 2 nA+nW #FLOPs for Reduction K µ · n… view at source ↗
Figure 5
Figure 5. Comparison between online and offline derived 4-bit quantization [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Comparison of (a) conventional dynamic outlier detection and (b) the [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: WAQ LUT-based GEMM computation scheme. § III-B presents the WAQ LUT-based GEMM scheme for efficient GEMMs with non-uniformly quantized weights and activations, and § III-C introduces the look-ahead computation and error compensation designs to handle activation outlier…
Figure 8
Figure 8. Figure 8: Overall architecture of the OASIS accelerator. [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Design of (a) the Index Counter and (b) the Clustering Unit. [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Orizuru architecture. (a) The overall architecture of Orizuru: two-fold binary trees with shared leaf nodes and comparison results at the last layer of [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Normalized throughput and energy consumption of OASIS and [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 12
Figure 12. Figure 12: Normalized throughput and energy consumption of OASIS and [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 13
Figure 13. Figure 13: Normalized throughput and energy consumption of OASIS and [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 17
Figure 17. Figure 17: Effects across calibration datasets and numbers of calibration samples [PITH_FULL_IMAGE:figures/full_fig_p013_17.png]
Figure 18
Figure 18. Figure 18: Breakdown of (a) memory traffic and (b) energy consumption of [PITH_FULL_IMAGE:figures/full_fig_p013_18.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 20 canonical work pages

  1. [1]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Introducing nvfp4 for efficient and accurate low-precision inference,

    E. Alvarez, O. Almog, E. Chung, S. Layton, D. Stosic, R. Krashin- sky, and K. Aubrey, “Introducing nvfp4 for efficient and accurate low-precision inference,” https://developer.nvidia.com/blog/introducing- nvfp4-for-efficient-and-accurate-low-precision-inference/, Jun 2025

    work page 2025

  3. [3]

    Quarot: Outlier-free 4-bit inference in rotated llms,

    S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, P. Cameron, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman, “Quarot: Outlier-free 4-bit inference in rotated llms,”arXiv preprint arXiv:2404.00456, 2024

    arXiv 2024

  4. [4]

    Piqa: Reasoning about physical commonsense in natural language,

    Y . Bisk, R. Zellers, J. Gao, Y . Choiet al., “Piqa: Reasoning about physical commonsense in natural language,” inProceedings of the AAAI conference on artificial intelligence, vol. 34, no. 05, 2020, pp. 7432– 7439

    2020

  5. [5]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language mod- els are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  6. [6]

    Boolq: Exploring the surprising difficulty of natural yes/no questions,

    C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,” inProceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers), 2019, pp. 2924–2936

    2019

  7. [7]

    Think you have solved question answering? try arc, the ai2 reasoning challenge,

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,”arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  8. [8]

    Nvidia rtx blackwell gpu architecture: Built for neural rendering,

    N. Corporation, “Nvidia rtx blackwell gpu architecture: Built for neural rendering,” NVIDIA Corporation, Tech. Rep. V1.1, 2025, white paper. [Online]. Available: https://images.nvidia.com/aem-dam/ Solutions/geforce/blackwell/nvidia-rtx-blackwell-gpu-architecture.pdf

    work page 2025

Show all 61 references
  1. [9]

    Documenting large webtext corpora: A case study on the colossal clean crawled corpus,

    J. Dodge, M. Sap, A. Marasovi ´c, W. Agnew, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner, “Documenting large webtext corpora: A case study on the colossal clean crawled corpus,”arXiv preprint arXiv:2104.08758, 2021

    2021 arXiv

  2. [11]

    Gptq: Accurate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,”arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  3. [12]

    Gpt4aigchip: Towards next-generation ai accelerator design automation via large language models,

    Y . Fu, Y . Zhang, Z. Yu, S. Li, Z. Ye, C. Li, C. Wan, and Y . C. Lin, “Gpt4aigchip: Towards next-generation ai accelerator design automation via large language models,” in2023 IEEE/ACM International Confer- ence on Computer Aided Design (ICCAD). IEEE, 2023, pp. 1–9

    2023

  4. [13]

    The language model evaluation harness,

    L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, “The languag...

    2024

  5. [14]

    Disaggregated machine learning via in-physics computing at radio frequency,

    Z. Gao, S. K. Vadlamani, K. Sulimany, D. Englund, and T. Chen, “Disaggregated machine learning via in-physics computing at radio frequency,”Science Advances, vol. 12, no. 2, p. eadz0817, 2026

    2026

  6. [15]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [16]

    A survey: Collaborative hardware and software design in the era of large language models,

    C. Guo, F. Cheng, Z. Du, J. Kiessling, J. Ku, S. Li, Z. Li, M. Ma, T. Molom-Ochir, B. Morriset al., “A survey: Collaborative hardware and software design in the era of large language models,”IEEE Circuits and Systems Magazine, vol. 25, no. 1, pp. 35–57, 2025

    2025

  8. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Biet al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,”arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  9. [18]

    Chateda: A large language model powered autonomous agent for eda,

    Z. He, H. Wu, X. Zhang, X. Yao, S. Zheng, H. Zheng, and B. Yu, “Chateda: A large language model powered autonomous agent for eda,” in2023 ACM/IEEE 5th Workshop on Machine Learning for CAD (MLCAD). IEEE, 2023, pp. 1–6

    2023

  10. [19]

    Kvquant: Towards 10 million context length llm inference with kv cache quantization,

    C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y . S. Shao, K. Keutzer, and A. Gholami, “Kvquant: Towards 10 million context length llm inference with kv cache quantization,”arXiv preprint arXiv:2401.18079, 2024

    2024 arXiv

  11. [20]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnieret al., “Mistral 7b,”arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  12. [21]

    Squeezellm: Dense-and-sparse quantization,

    S. Kim, C. Hooper, A. Gholami, Z. Dong, X. Li, S. Shen, M. W. Mahoney, and K. Keutzer, “Squeezellm: Dense-and-sparse quantization,” arXiv preprint arXiv:2306.07629, 2023

    2023 arXiv

  13. [22]

    Scaling laws for precision,

    T. Kumar, Z. Ankner, B. F. Spector, B. Bordelon, N. Muennighoff, M. Paul, C. Pehlevan, C. R ´e, and A. Raghunathan, “Scaling laws for precision,”arXiv preprint arXiv:2411.04330, 2024

    2024 arXiv

  14. [23]

    Maeri: Enabling flexible dataflow mapping over dnn accelerators via reconfigurable intercon- nects,

    H. Kwon, A. Samajdar, and T. Krishna, “Maeri: Enabling flexible dataflow mapping over dnn accelerators via reconfigurable intercon- nects,”ACM Sigplan Notices, vol. 53, no. 2, pp. 461–475, 2018

    2018

  15. [24]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th Symposium on Operating Systems Principles, 2023, pp. 611–626

    2023

  16. [25]

    Fast and efficient 2-bit llm inference on gpu: 2/4/16-bit in a weight matrix with asynchronous dequantization,

    J. Li, J. Xu, S. Li, S. Huang, J. Liu, Y . Lian, and G. Dai, “Fast and efficient 2-bit llm inference on gpu: 2/4/16-bit in a weight matrix with asynchronous dequantization,”arXiv preprint arXiv:2311.16442, 2023

    2023 arXiv

  17. [26]

    Dramsim3: A cycle-accurate, thermal-capable dram simulator,

    S. Li, Z. Yang, D. Reddy, A. Srivastava, and B. Jacob, “Dramsim3: A cycle-accurate, thermal-capable dram simulator,”IEEE Computer Architecture Letters, vol. 19, no. 2, pp. 106–109, 2020

    2020

  18. [27]

    Cacti- p: Architecture-level modeling for sram-based structures with advanced leakage reduction techniques,

    S. Li, K. Chen, J. H. Ahn, J. B. Brockman, and N. P. Jouppi, “Cacti- p: Architecture-level modeling for sram-based structures with advanced leakage reduction techniques,” in2011 IEEE/ACM International Confer- ence on Computer-Aided Design (ICCAD). IEEE, 2011, pp. 694–701

    2011

  19. [28]

    Duquant: Distributing outliers via dual transformation makes stronger quantized llms,

    H. Lin, H. Xu, Y . Wu, J. Cui, Y . Zhang, L. Mou, L. Song, Z. Sun, and Y . Wei, “Duquant: Distributing outliers via dual transformation makes stronger quantized llms,”Advances in Neural Information Processing Systems, vol. 37, pp. 87 766–87 800, 2025

    2025

  20. [29]

    Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,”Proceedings of Machine Learning and Systems, vol. 6, pp. 87–100, 2024

    2024

  21. [30]

    Qserve: W4a8kv4 quantization and system co-design for efficient llm serving,

    Y . Lin, H. Tang, S. Yang, Z. Zhang, G. Xiao, C. Gan, and S. Han, “Qserve: W4a8kv4 quantization and system co-design for efficient llm serving,”arXiv preprint arXiv:2405.04532, 2024

    2024 arXiv

  22. [31]

    Llm-fp4: 4-bit floating-point quantized transformers,

    S.-y. Liu, Z. Liu, X. Huang, P. Dong, and K.-T. Cheng, “Llm-fp4: 4-bit floating-point quantized transformers,”arXiv preprint arXiv:2310.16836, 2023

    2023 arXiv

  23. [32]

    Micromix: Efficient mixed-precision quantization with microscaling formats for large lan- guage models,

    W. Liu, H. Meng, Y . Luo, P. Zhang, and X. Ma, “Micromix: Efficient mixed-precision quantization with microscaling formats for large lan- guage models,”arXiv preprint arXiv:2508.02343, 2025

    2025

  24. [33]

    Spinquant–llm quantization with learned rotations,

    Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Krishnamoorthi, V . Chandra, Y . Tian, and T. Blankevoort, “Spinquant–llm quantization with learned rotations,”arXiv preprint arXiv:2405.16406, 2024

    2024 arXiv

  25. [34]

    Some methods for classification and analysis of mul- tivariate observations,

    J. MacQueen, “Some methods for classification and analysis of mul- tivariate observations,” inProceedings of 5-th Berkeley Symposium on Mathematical Statistics and Probability/University of California Press, 1967

    1967

  26. [35]

    The penn treebank: Anno- tating predicate argument structure,

    M. Marcus, G. Kim, M. A. Marcinkiewicz, R. MacIntyre, A. Bies, M. Ferguson, K. Katz, and B. Schasberger, “The penn treebank: Anno- tating predicate argument structure,” inHuman Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994

    1994

  27. [36]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,”arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  28. [37]

    Lut tensor core: A software-hardware co-design for lut-based low-bit llm inference,

    Z. Mo, L. Wang, J. Wei, Z. Zeng, S. Cao, L. Ma, N. Jing, T. Cao, J. Xue, F. Yanget al., “Lut tensor core: A software-hardware co-design for lut-based low-bit llm inference,” inProceedings of the 52nd Annual International Symposium on Computer Architecture, 2025, pp. 514–528

    2025

  29. [38]

    Flexagon: A multi-dataflow sparse-sparse matrix multiplication accelerator for efficient dnn processing,

    F. Mu ˜noz-Mart´ınez, R. Garg, M. Pellauer, J. L. Abell ´an, M. E. Aca- cio, and T. Krishna, “Flexagon: A multi-dataflow sparse-sparse matrix multiplication accelerator for efficient dnn processing,” inProceedings of the 28th ACM International Conference on Architectural Suppo...

    2023

  30. [39]

    Tensor core performance: The ultimate guide,

    NVIDIA, “Tensor core performance: The ultimate guide,” NVIDIA, Tech. Rep., 2019

    2019

  31. [40]

    Nvidia a100 tensor core gpu architecture,

    ——, “Nvidia a100 tensor core gpu architecture,” NVIDIA, Tech. Rep., 2020

    2020

  32. [41]

    Nvidia turing gpu architecture whitepaper,

    NVIDIA Corporation, “Nvidia turing gpu architecture whitepaper,” NVIDIA Corporation, Tech. Rep. 87 pages, 2018. [Online]. Available: https://images.nvidia.com/aem-dam/en-zz/Solutions/design- visualization/technologies/turing-architecture/NVIDIA-Turing- Architecture-Whitepaper.pdf

    2018

  33. [42]

    Figlut: An energy-efficient accelerator design for fp-int gemm using look-up tables,

    G. Park, H. Kwon, J. Kim, J. Bae, B. Park, D. Lee, and Y . Lee, “Figlut: An energy-efficient accelerator design for fp-int gemm using look-up tables,” in2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2025, pp. 1098–1111

    2025

  34. [43]

    Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models,

    G. Park, B. Park, M. Kim, S. Lee, J. Kim, B. Kwon, S. J. Kwon, B. Kim, Y . Lee, and D. Lee, “Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models,” arXiv preprint arXiv:2206.09557, 2022

    2022 arXiv

  35. [44]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antigaet al., “Pytorch: An imperative style, high-performance deep learning library,”Advances in neural information processing systems, vol. 32, 2019

    2019

  36. [45]

    The spectrum of the fisher information matrix of a single-hidden-layer neural network,

    J. Pennington and P. Worah, “The spectrum of the fisher information matrix of a single-hidden-layer neural network,”Advances in neural information processing systems, vol. 31, 2018

    2018

  37. [46]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remezet al., “Code llama: Open foundation models for code,”arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  38. [47]

    Winogrande: An adversarial winograd schema challenge at scale,

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “Winogrande: An adversarial winograd schema challenge at scale,”Communications of the ACM, vol. 64, no. 9, pp. 99–106, 2021

    2021

  39. [48]

    From high-level deep neural models to fpgas,

    H. Sharma, J. Park, D. Mahajan, E. Amaro, J. K. Kim, C. Shao, A. Mishra, and H. Esmaeilzadeh, “From high-level deep neural models to fpgas,” in2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 2016, pp. 1–12

    2016

  40. [49]

    Using tournament trees to sort,

    A. A. Stepanov and A. Kershenbaum, “Using tournament trees to sort,” Center for Advanced Technology in Telecommunications, Polytechnic University of New York, Tech. Rep. 86-13, 1986

    1986

  41. [50]

    Flatquant: Flatness matters for llm quantization,

    Y . Sun, R. Liu, H. Bai, H. Bao, K. Zhao, Y . Li, J. Hu, X. Yu, L. Hou, C. Yuanet al., “Flatquant: Flatness matters for llm quantization,”arXiv preprint arXiv:2410.09426, 2024

    2024 arXiv

  42. [51]

    Crystal: Illuminating llm abilities on language and code,

    T. Tao, J. Li, B. Tan, H. Wang, W. Marshall, B. M. Kanakiya, J. Hestness, N. Vassilieva, Z. Shen, E. P. Xinget al., “Crystal: Illuminating llm abilities on language and code,”arXiv preprint arXiv:2411.04156, 2024

    2024 arXiv

  43. [52]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  44. [53]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  45. [54]

    Training llms with mxfp4,

    A. Tseng, T. Yu, and Y . Park, “Training llms with mxfp4,”arXiv preprint arXiv:2502.20586, 2025

    2025 arXiv

  46. [55]

    Spatten: Efficient sparse attention architecture with cascade token and head pruning,

    H. Wang, Z. Zhang, and S. Han, “Spatten: Efficient sparse attention architecture with cascade token and head pruning,” in2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2021, pp. 97–110

    2021

  47. [56]

    Transformers: State- of-the-art natural language processing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowiczet al., “Transformers: State- of-the-art natural language processing,” inProceedings of the 2020 conference on empirical methods in natural language processing: system demo...

    2020

  48. [57]

    Block- wise mixed-precision quantization: Enabling high efficiency for practical reram-based dnn accelerators,

    X. Wu, E. Hanson, N. Wang, Q. Zheng, X. Yang, H. Yang, S. Li, F. Cheng, P. P. Pande, J. R. Doppa, K. Chakrabarty, and H. Li, “Block- wise mixed-precision quantization: Enabling high efficiency for practical reram-based dnn accelerators,”IEEE Transactions on Computer-Aided Desi...

    2024

  49. [58]

    Smoothquant: Accurate and efficient post-training quantization for large language models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 38 087–38 099

    2023

  50. [59]

    Hel- laswag: Can a machine really finish your sentence?

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hel- laswag: Can a machine really finish your sentence?”arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  51. [60]

    Lq-nets: Learned quantization for highly accurate and compact deep neural networks,

    D. Zhang, J. Yang, D. Ye, and G. Hua, “Lq-nets: Learned quantization for highly accurate and compact deep neural networks,” inProceedings of the European conference on computer vision (ECCV), 2018, pp. 365– 382

    2018

  52. [61]

    Opt: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Linet al., “Opt: Open pre-trained transformer language models,”arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  53. [62]

    Atom: Low-bit quantization for efficient and accurate llm serving,

    Y . Zhao, C.-Y . Lin, K. Zhu, Z. Ye, L. Chen, S. Zheng, L. Ceze, A. Krishnamurthy, T. Chen, and B. Kasikci, “Atom: Low-bit quantization for efficient and accurate llm serving,”Proceedings of Machine Learning and Systems, vol. 6, pp. 196–209, 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.