Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

LiquidGEMM reduces W4A8 dequantization to two arithmetic instructions per four elements, unlocking up to 2.90x kernel speedup.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A W4A8 GEMM kernel using shift-based overflow-safe dequantization and a hardware-scheduled pipeline reports up to 2.9x speedup over prior W4A8 kernels.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Solid kernel paper with a genuinely clever dequantization trick; the missing accuracy numbers are the one thing that keeps it from being citable as a serving system. the 3 major comments →

arxiv 2509.01229 v1 pith:ZMBCRUQ5 submitted 2025-09-01 cs.DC cs.AIcs.LG

LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving

Huanqi Hu , Bowen Xiao , Shixuan Sun , Jianian Yin , Zhexi Zhang , Xiang Luo , Chengquan Jiang , Weiqi Xu
show 3 more authors
Xiaoying Jia Xin Liu Minyi Guo
This is my paper
classification cs.DC cs.AIcs.LG
keywords W4A8 quantizationGEMM kernelLLM servingdequantizationTensor Coresimplicit pipelineinteger quantizationGPU kernel optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that W4A8 quantization—4-bit weights, 8-bit activations—can finally deliver its theoretical speed advantage in LLM serving, but only if the dequantization step stops competing with the GPU's fast Tensor Cores. It claims that prior W4A8 kernels spend so many instructions recovering 8-bit values from packed 4-bit weights on the slower CUDA cores that the quantization advantage disappears, sometimes making W4A8 slower than W8A8. LiquidGEMM's two contributions are LiquidQuant, a rotation-based quantization scheme that makes dequantization overflow-safe in exactly two arithmetic instructions (IMAD and XOR) per four elements, and an implicit fine-grained pipeline that overlaps weight loading, dequantization, and matrix multiply without explicit synchronization. If correct, dequantization is no longer the bottleneck, and W4A8 becomes practical for production serving, with reported kernel speedups of up to 2.90x over the leading W4A8 kernel and 1.12–1.63x over TensorRT-LLM's quantized kernels. The paper also reports up to 4.94x end-to-end system-level speedup, while deferring the model-accuracy evidence that would make the whole system viable.

Core claim

The paper's central claim is that the dequantization step, not the tensor-core MMA, is what makes W4A8 GEMM underperform, and that both halves of the problem—the arithmetic and the scheduling—can be redesigned to remove it. LiquidQuant re-quantizes weights so that decoding UINT4 back to INT8 reduces to (q_u4 * s_u8 + a) XOR 0x80, a multiply-add plus a bit flip that stays in the UINT8 range by construction, costing two hardware instructions per four elements instead of the instruction-heavy overflow handling used by prior kernels. The implicit fine-grained pipeline assigns one warp group to load weights and several compute warp groups to dequantize and immediately run MMA on them, letting har

What carries the argument

LiquidQuant's sweet dequantization identity (q_u4 * s_u8 + a) XOR 0x80, where a = 2^7 + min(Q_i8) is precomputed offline, carries the arithmetic: it turns an overflow-prone signed add into an unsigned multiply-add plus a most-significant-bit flip, provably staying inside UINT8 and executable by native IMAD and XOR. The implicit fine-grained pipeline (ImFP) carries the execution: a single-producer, multiple-consumer model where one load warp group feeds shared memory and several compute warp groups each dequantize and run MMA, using hardware task arbitration instead of software barriers to overlap loading, dequantization, and tensor-core work.

Load-bearing premise

The argument's load-bearing premise is that LiquidQuant's W4A8 models keep accuracy close to previous W4A8 methods; the paper asserts this but withholds the accuracy tables (Section 7.1), so the kernel speedups only matter if that assertion holds.

What would settle it

Run the deferred accuracy evaluation: measure WikiText2 perplexity and the listed zero-shot tasks for LQQ-quantized LLaMA2-7B, LLaMA2-70B, and Mixtral-8x7B against QServe's W4A8 numbers; if perplexity rises significantly or zero-shot accuracy drops, the serving claim collapses. Separately, disassemble the shipped kernel to confirm dequantization compiles to one IMAD and one XOR per four elements and that no software barriers appear in the main loop.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • W4A8 GEMM no longer needs a dedicated dequantization warp group; the same warp can dequantize and directly feed Tensor Cores.
  • The kernel scales better at large batch sizes than prior W4A8 kernels, because dequantization is overlapped rather than serialized.
  • In memory-bound small-batch scenarios, W4A8's lower memory traffic can beat W8A8 and FP8; in compute-bound large-batch scenarios it can match or beat them.
  • System-level serving throughput improves up to 4.94x over QServe and up to 1.63x over TensorRT-LLM, with the largest gains on large models like LLaMA2-70B.
  • LiquidQuant is orthogonal to activation-smoothing and accuracy-oriented quantization methods, so it can be combined with them rather than competing.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the deferred accuracy report confirms that LQQ matches prior W4A8 accuracy, the kernel speedups translate directly into production gains; if it does not, the kernel is fast but the serving system is not viable as claimed.
  • The same bit-flip dequantization trick should generalize to other asymmetric low-bit formats, such as W3A8 or per-group UINT4 on other GPUs, because it relies only on two's complement arithmetic and a bounded protective range.
  • The implicit pipeline style—hardware-arbitrated task consumption instead of software barriers—could be applied to other mixed-precision kernels where a cheap operand transformation sits between load and MMA.
  • The cost model implies that as Tensor Core throughput grows faster than memory bandwidth, keeping per-element dequantization cost small becomes more important; LiquidQuant's constant two-instruction cost lowers the batch size at which W4A8 becomes compute-bound.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents LiquidGEMM, a W4A8 GEMM kernel for LLM serving on Hopper-class GPUs. It proposes LiquidQuant (LQQ), a two-level quantization scheme that shifts INT8 weights into the UINT8 domain before quantizing to UINT4, and dequantizes with two native instructions (IMAD and XOR) per four elements. A mathematical proof is given that the dequantization computation remains overflow-free within UINT8. The paper also designs an implicit fine-grained pipeline (ImFP) with one load warp group and two compute warp groups to overlap TMA loading, CUDA-core dequantization, and Tensor-Core MMA without explicit synchronization, and a dual-MMA packed SMEM layout that makes LDS.128 efficient for 4-bit weights. Experiments on H800 compare LiquidGEMM with QServe and TensorRT-LLM kernels and report up to 2.90x kernel speedup and up to 4.94x end-to-end system-level speedup, while the accuracy evaluation of LQQ is deferred to a future report.

Significance. If the kernel results are reproducible, the dequantization scheme is a genuine contribution: the overflow-freedom proof in Section 4 is self-contained, the IMAD+XOR formulation directly attacks the identified CUDA-core bottleneck, and the ablation study cleanly separates the effects of LQQ, ExCP, and ImFP. The cost model also provides transparent, falsifiable thresholds on per-element instruction cost. However, the paper's system-level claims are currently incomplete: the accuracy of LQQ is asserted but not measured, and the abstract's 4.94x end-to-end speedup is not traceable to any experiment in Section 7. The kernel may be fast, but the paper has not yet established that it enables a usable W4A8 serving system.

major comments (3)
  1. [Section 7.1, Experiment Roadmap] The paper states 'Results show that LQQ preserves accuracy' and then defers all perplexity and zero-shot results to a future technical report. This is load-bearing for the abstract's end-to-end serving claims and for Table 1: if LQQ's quantization error is materially worse than QServe's, the 'up to 4.94x end-to-end speedup' does not establish a practically viable W4A8 serving system. Please include the accuracy tables (WikiText2 perplexity and zero-shot accuracies for the models cited) in this manuscript, or explicitly rescope the claims to kernel-level performance only.
  2. [Abstract and Section 9] The abstract claims 'up to 4.94x end-to-end system-level speedup', but no experiment in Section 7 supports this number. Table 1 reports speedups of 0.98x to 1.63x relative to the best baseline, and Section 7.2 reports a 1.13-1.98x speedup over LiquidServe/wo. The 4.94x figure appears in the abstract and conclusion only. Please identify the specific experiment and baseline that produce this number, or correct the claim. Without traceability, the headline result is unsupported.
  3. [Section 4, Eq. (7)] The scale factor s_u8 is not defined as an integer, yet the dequantization formula (Eq. 12) relies on an integer multiply for IMAD and the proof relies on s_u8 <= 16. The text first states s_u8 <= floor((119-(-119))/15) = 16, but floor(238/15) = 15; the value 16 comes from rounding to nearest. Please specify the exact integer rounding rule for s_u8 (e.g., s_u8 = round(max(Q_u8)/15) or floor(...)), and reconcile the bound accordingly. This matters both for the overflow-free guarantee and for the quantization error of LQQ.
minor comments (5)
  1. [Abstract and Section 5.3] The abstract says 'two arithmetic instructions per four elements', but Section 5.3 says that including the unpacking step, eight elements are dequantized with seven instructions. Please clarify whether the 'two instructions' counts only the IMAD+XOR dequantization after unpacking or the entire per-element cost; the current wording is misleading.
  2. [Section 4, Eq. (10)] The notation in Eq. (10) uses floor/ceil-style rounding on q_u/s_u8, but the relationship between Q_u4, Q_u8, and s_u8 in Eq. (7) should be stated more precisely. In particular, state explicitly that quantization uses rounding to nearest integer and that s_u8 is the rounded integer scale.
  3. [Section 7.2, Table 1] QServe's throughput for Mistral-7B and Yi-34B is much lower than for other systems (e.g., 5,240 and 5,361 tokens/s). Please clarify whether QServe supports these models in its public implementation or whether its W4A8 path is unoptimized for them; otherwise the comparison may overstate LiquidServe's advantage on those rows.
  4. [Section 3.1 / Figure 1] The motivation and cost model quote H100 hardware metrics, but all experiments are on H800. H800 has different NVLink and some different hardware characteristics; please state whether the Tensor Core and CUDA Core throughputs used in the cost model are the same on H800, or provide H800-specific numbers.
  5. [Section 7.3, Figure 12] The GEMM latency results are stated as averages over five runs, but no error bars or per-run variance are reported. Please add a sentence on variance or include error bars, especially for the claimed 2.90x speedup at batch size 256.

Circularity Check

0 steps flagged

No significant circularity: LQQ's overflow-safe dequantization is derived and proved in-paper, and the kernel benchmarks are external measurements.

full rationale

The central derivation chain is self-contained. Section 4 defines LQQ quantization (Eq. 7), derives the dequantization identity (Eqs. 9-10), proves the UINT8-range bound under the adopted Q_i8∈[-119,119] assumption (Eq. 11), and reduces the result to bQ_i8 = (Q_u4*s_u8 + a) XOR 0x80 (Eq. 12). The hardware claim (IMAD + XOR per four elements) follows from this equation and the instruction mapping in Section 5.3; it is not a fitted parameter renamed as a prediction. The cost model in Section 3 uses measured hardware throughputs as inputs, but it is used to derive design thresholds (e.g., alpha≤5.07) and to explain the observed QServe bottleneck; the later speedups are measured against external baselines, not generated by the model. The only adoption from prior work is the protective quantization range Q_i8∈[-119,119] and the unpacking trick, both cited to QServe [15]; this is external support with stated assumptions, not a self-citation chain, and it does not by itself imply the target result. The paper does contain one explicit missing-support assertion: Section 7.1 says 'Results show that LQQ preserves accuracy. Due to space constraints, detailed results will be released in a full-version technical report.' This is a genuine correctness/verifiability gap for the end-to-end usability claim (the kernel could be fast while LQQ models are too degraded), but it is not a circularity: the kernel performance and system throughput numbers are independent measurements, and the accuracy claim is an unverified premise rather than an input that the derivation reduces back to. No self-definitional, fitted-input, or uniqueness-importation pattern is present. Therefore circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new particles, mediators, forces, or physical entities are introduced. 'LiquidQuant' and 'ImFP' are methods, not entities.

free parameters (4)
  • weight quantization group size = 64
    Chosen as default, not compared in the paper; affects accuracy and dequantization granularity.
  • number of Compute WGs per thread block = 2
    Hand-selected design choice; no sensitivity study reported.
  • GEMM tile sizes (M_t, N_t, K_t)
    Not specified in the paper; presumably tuned for H800. The kernel performance claim depends on these values.
  • smooth scale (from OutlierSuppression+ grid search) = not reported
    Fitted to model data during offline quantization; follows prior work, not the kernel's central derivation.
axioms (4)
  • domain assumption Hardware throughput metrics in Figure 1 (Tensor Core INT8 ~1979 TOPS, CUDA Core ~33.5 TOPS on H100) are accurate for cost model analysis.
    Used in Section 3.2-3.3 to derive alpha thresholds; errors would shift the claimed bottleneck analysis but not the measured speedups.
  • domain assumption The cost model (Eq. 6) with min(M_t, M) approximates the dominant costs of the main loop; epilogue and pipeline fill/drain are negligible.
    Stated in Section 3.2; used to motivate the design. Not directly used for the headline speedups.
  • domain assumption QServe's protective quantization range [-119, 119] preserves sufficient accuracy when adopted by LQQ.
    LQQ inherits this range from QServe [15]; the paper does not re-validate accuracy.
  • standard math The two's complement congruence property i ≡ j (mod 2^8) means XOR with 0x80 correctly recovers INT8 from UINT8 representation.
    This is the mathematical core of LQQ and is proven in Section 4.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving." pith.science (2026). https://pith.science/paper/ZMBCRUQ5

@misc{pith2026250901229,
  author       = {Pith},
  title        = {Pith review of: LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZMBCRUQ5}},
  note         = {Machine review of arXiv:2509.01229}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Quantization is a critical technique for accelerating LLM inference by reducing memory footprint and improving computational efficiency. Among various schemes, 4-bit weight and 8-bit activation quantization (W4A8) offers a strong balance between accuracy and performance. However, existing W4A8 GEMM kernels fall short in practice due to inefficient dequantization on CUDA Cores, which cannot keep pace with the high throughput of Tensor Cores. In this paper, we present LiquidGEMM, a hardware-efficient W4A8 GEMM kernel for efficient LLM serving. LiquidGEMM designs two key techniques: LiquidQuant, a hardware-efficient quantization method that enables fast, overflow-safe dequantization using just two arithmetic instructions per four elements; and an implicit fine-grained pipeline that fully overlaps weight loading, dequantization, and MMA across warp groups without software synchronization or redundant memory traffic. Experimental results show that LiquidGEMM achieves up to 2.90x speedup over state-of-the-art W4A8 kernels and up to 4.94x end-to-end system-level speedup. Compared to various quantized GEMM kernels in NVIDIA TensorRT-LLM, LiquidGEMM delivers 1.12-1.63x performance gains, and achieves up to 1.63x system-level speedup.

Figures

Figures reproduced from arXiv: 2509.01229 by Bowen Xiao, Chengquan Jiang, Huanqi Hu, Jianian Yin, Minyi Guo, Shixuan Sun, Weiqi Xu, Xiang Luo, Xiaoying Jia, Xin Liu, Zhexi Zhang.

Figure 1
Figure 1. Figure 1: Key performance metrics of NVIDIA A100 and H100 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of GEMM on GPUs, where 𝑖, 𝑗, 𝑙 denote loop iterations along the 𝑀, 𝑁, 𝐾 dimensions, respectively. including FlashAttention [6] for attention computation and Page￾dAttention [12] for KV cache management. Experimental results demonstrate that LiquidGEMM achieves up to 2.90x speedup over the state-of-the-art W4A8 kernel [15], and leads to up to 4.94x end￾to-end system-level speedup. Compared with var… view at source ↗
Figure 4
Figure 4. Figure 4: Time breakdown of inference for input lengths 128 [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: GEMM latency on a single transformer layer with [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of Explicit Coarse-Grained Pipeline [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of conventional memory layout and [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Overview of dataflows in our LLM serving system [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: Comparison of token generation throughput [PITH_FULL_IMAGE:figures/full_fig_p010_11.png] view at source ↗
Figure 10
Figure 10. Figure 10: Time breakdown for processing one decoding layer [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: Comparison of GEMM latency on the FFN layer [PITH_FULL_IMAGE:figures/full_fig_p010_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Ablation study of LiquidGEMM by first enabling [PITH_FULL_IMAGE:figures/full_fig_p011_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference

    stat.ML 2026-05 unverdicted novelty 7.0

    MSD eliminates dequantization from the GEMM path by decomposing BF16 activations into multiple low-precision parts that multiply directly with INT8 or MXFP4 weights, achieving near-16 effective bits for INT8 and 6.6 f...

  2. LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers

    cs.LG 2025-09 unverdicted novelty 4.0

    PreScope combines a layer-aware activation predictor, cross-layer prefetch scheduling, and asynchronous I/O to deliver 141% higher throughput and 74.6% lower latency for MoE inference on legacy hardware.

Reference graph

Works this paper leans on

37 extracted references · 10 canonical work pages · cited by 2 Pith papers · 2 internal anchors

  1. [1]

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pash- mina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hens- man. 2024. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456 (2024)

    Pith/arXiv arXiv 2024

  2. [2]

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , Vol. 34. 7432–7439

    work page 2020

  3. [3]

    Yelysei Bondarenko, Riccardo Del Chiaro, and Markus Nagel. 2024. Low-Rank Quantization-Aware Training for LLMs. arXiv preprint arXiv:2406.06385 (2024)

    Pith/arXiv arXiv 2024

  4. [4]

    Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. 2024. Efficientqat: Efficient quantization-aware training for large language models. arXiv preprint arXiv:2407.11062 (2024)

    Pith/arXiv arXiv 2024

  5. [5]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 (2018)

    Pith/arXiv arXiv 2018

  6. [6]

    Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 (2023). 11 Huanqi Hu et al

    Pith/arXiv arXiv 2023

  7. [7]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems 35 (2022), 30318–30332

    work page 2022

  8. [8]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 (2022)

    Pith/arXiv arXiv 2022

  9. [9]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schel- ten, Alex Vaughan, et al . 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    Pith/arXiv arXiv 2024

  10. [10]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. arXiv:2310.068...

    Pith/arXiv arXiv 2023

  11. [11]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al . 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024)

    Pith/arXiv arXiv 2024

  12. [12]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles . 611–626

    2023

  13. [13]

    Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. 2025. Duquant: Distributing outliers via dual transformation makes stronger quantized llms. Advances in Neural Information Processing Systems 37 (2025), 87766–87800

    work page 2025

  14. [14]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. Proceedings of Machine Learning and Systems 6 (2024), 87–100

    work page 2024

  15. [15]

    Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2024. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. arXiv preprint arXiv:2405.04532 (2024)

    Pith/arXiv arXiv 2024

  16. [16]

    Lian Liu, Haimeng Ren, Long Cheng, Zhaohui Xu, Yudong Pan, Mengdi Wang, Xi- aowei Li, Yinhe Han, and Ying Wang. 2024. COMET: Towards Partical W4A4KV4 LLMs Serving. arXiv preprint arXiv:2410.12168 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  17. [17]

    Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. 2023. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888 (2023)

    Pith/arXiv arXiv 2023

  18. [18]

    Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2024. SpinQuant–LLM quantization with learned rotations. arXiv preprint arXiv:2405.16406 (2024)

    Pith/arXiv arXiv 2024

  19. [19]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 (2016)

    Pith/arXiv arXiv 2016

  20. [20]

    NVIDIA. 2023. TensorRT-LLM: A TensorRT Toolbox for Optimized Large Language Model Inference. https://github.com/NVIDIA/TensorRT-LLM

    work page 2023

  21. [21]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale.Commun. ACM 64, 9 (2021), 99–106

    2021

  22. [22]

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. 2024. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems 37 (2024), 68658–68685

    work page 2024

  23. [23]

    Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2023. Omniquant: Omni- directionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137 (2023)

    Pith/arXiv arXiv 2023

  24. [24]

    Xuan Shen, Zhenglun Kong, Changdi Yang, Zhaoyang Han, Lei Lu, Peiyan Dong, Cheng Lyu, Chih-hsiang Li, Xuehang Guo, Zhihao Shu, et al. 2024. EdgeQAT: En- tropy and Distribution Guided Quantization-Aware Training for the Acceleration of Lightweight LLMs on the Edge. arXiv preprint arXiv:2402.10787 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  25. [25]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    Pith/arXiv arXiv 2023

  26. [26]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    Pith/arXiv arXiv 2023

  27. [27]

    Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. 2023. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453 (2023)

    Pith/arXiv arXiv 2023

  28. [28]

    Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. 2023. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145 (2023)

    Pith/arXiv arXiv 2023

  29. [29]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning . PMLR, 38087–38099

    2023

  30. [30]

    Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. 2024. Onebit: Towards extremely low-bit large language models. arXiv preprint arXiv:2402.11295 (2024)

    Pith/arXiv arXiv 2024

  31. [31]

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al . 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652 (2024)

    Pith/arXiv arXiv 2024

  32. [32]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung- Gon Chun. 2022. Orca: A distributed serving system for{Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521–538

    work page 2022

  33. [33]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi

  34. [34]

    Ying Zhang, Peng Zhang, Mincong Huang, Jingyang Xiang, Yujie Wang, Chao Wang, Yineng Zhang, Lei Yu, Chuan Liu, and Wei Lin. 2024. QQQ: Qual- ity Quattuor-Bit Quantization for Large Language Models. arXiv preprint arXiv:2406.09904 (2024)

    Pith/arXiv arXiv 2024

  35. [35]

    Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024. Atom: Low- bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems 6 (2024), 196–209

    work page 2024

  36. [36]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving.arXiv preprint arXiv:2401.09670 (2024). 12

    Pith/arXiv arXiv 2024

  37. [2019]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 (2019)

    Pith/arXiv arXiv 1905

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.