pith. sign in

arxiv: 2606.11244 · v1 · pith:TXJNI7UXnew · submitted 2026-06-04 · 💻 cs.AR · cs.AI

SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Serving

Hongyuan Liu , Yawei Li , Zhiqiang Que , Qinli Yang , Junming Shao , Guosheng Hu This is my paper

Pith reviewed 2026-06-27 23:02 UTC · model grok-4.3

classification 💻 cs.AR cs.AI
keywords LLM quantizationpost-training compensationadaptive error recoverylow-bit inferencekernel fusionper-token gatingmodel serving systems
0
0 comments X

The pith

SPEAR recovers 56-75% of the perplexity gap from 4-bit LLM quantization by applying input-dependent error compensation only at sensitive layers.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper identifies that quantization error varies sharply with each input token, so static correction methods waste effort on easy tokens while leaving hard ones uncorrected. SPEAR counters this by inserting small error compensators that activate selectively via per-token gates, but only in layers flagged by a diagnostic that measures sensitivity through representation similarity and entropy. These additions are kept efficient by fusing the extra work into existing low-bit matrix operations and using a scheduler that respects service-level objectives. If the approach holds, 4-bit models can deliver output quality much nearer to full precision while retaining their memory and speed advantages for deployment.

Core claim

SPEAR introduces lightweight Error Compensators modulated by per-token gates and places them only at the most error-sensitive layers identified through a CKA-guided entropy-aware diagnostic. This focuses a small parameter budget where it is most effective. Efficient deployment of ECs is achieved through adaptive kernel-fusion dispatch that combines an epilogue-integrated peer-reduction kernel with P2P dual-write to fuse the post-EC computation into low-bit GEMMs, plus an SLO-constrained EC-aware scheduler. Across challenging per-channel quantization settings, SPEAR recovers 56-75% of the perplexity gap between W4 and FP16 while adding less than 1% model memory overhead and maintaining latenc

What carries the argument

Lightweight Error Compensators (ECs) modulated by per-token gates, placed via CKA-guided entropy-aware diagnostic and deployed through adaptive kernel-fusion dispatch with epilogue-integrated peer-reduction and P2P dual-write.

If this is right

  • 4-bit quantized LLMs can achieve perplexity values substantially closer to FP16 without increasing model memory footprint beyond 1%.
  • Serving systems can maintain the low latency of existing 4-bit deployments while reducing the quality penalty on difficult inputs.
  • Compensation effort can be concentrated on a small number of layers and activated only when needed, avoiding uniform overhead across all tokens.
  • The same placement and fusion strategy supports predictable performance under service-level objectives even when input difficulty varies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The per-token diagnostic might be reusable to decide where to apply other forms of dynamic correction, such as speculative decoding triggers.
  • Extending the same adaptive logic to 3-bit or 2-bit quantization could test whether the recovery percentage scales with the initial error magnitude.
  • In multi-tenant serving, the scheduler's awareness of EC cost could be used to prioritize batches with lower expected compensation overhead.

Load-bearing premise

That the variation in quantization error across tokens is large enough for per-token gating to yield a net quality gain without introducing synchronization or latency costs that grow with model size.

What would settle it

A benchmark run on standard tensor-parallel hardware showing that the per-token gating and fused kernels increase end-to-end latency by more than a few percent relative to baseline 4-bit serving would falsify the efficiency claim.

Figures

Figures reproduced from arXiv: 2606.11244 by Guosheng Hu, Hongyuan Liu, Junming Shao, Qinli Yang, Yawei Li, Zhiqiang Que.

Figure 1
Figure 1. Figure 1: Per-token similarity between FP16 and 4- [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: SPEAR framework. Algorithm-wise, the compensation pipeline (left) identifies the modules most [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Architecture of Error Compensator (EC): the low-rank coordinates Ax are modulated by an input-dependent gate γ(Ax) before projection by B, so the effective compensation adapts per token. ⊙ and ⊕ denote element-wise multiplication and addition, respectively. We therefore introduce the Error Compensator (EC), an input-adaptive low-rank compensation mod￾ule that dynamically modulates compensation in the rank-… view at source ↗
Figure 4
Figure 4. Figure 4: Per-module quantization damage on Llama-3.2-1B and Llama-2-7B under RTN (Round to Nearest, [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Phase-aware adaptive kernel fusion dispatch. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: Split vs. fused kernel execution. Top: Naive EC requires multiple separate kernel launches whose launch gaps (grey) dominate over actual compute (red). Bottom: Fused execution embeds the full EC chain into the 4-bit GEMM epilogue, collapsing the layer into a single kernel and eliminating inter-kernel overhead. However, the optimal fusion strategy depends on the serving phase. During decode (Batch size M=1)… view at source ↗
Figure 7
Figure 7. Figure 7: The EC path, therefore, requires an explicit cross-GPU synchronization before the remaining EC computation proceeds. This exposed synchronization is particularly ineffi￾cient during decode, where execution is highly latency￾sensitive. Although the communicated EC activation is low-rank and small in bandwidth, the EC path still incurs separate kernel launches, NCCL scheduling, and synchronization overhead. … view at source ↗
Figure 8
Figure 8. Figure 8: reports end-to-end per-token decode latency under single-token generation (M=1) for four configu￾rations: FP16 cuBLAS, W4 MARLIN, naive W4+EC deployment, and SPEAR’s optimized serving stack. Naively inserting ECs makes low-bit decode imprac￾tical. Across 1B, 3B, and 7B models, the unfused W4+EC pipeline increases decode latency by roughly 5× over plain W4 MARLIN, largely eliminating the throughput advantag… view at source ↗
Figure 9
Figure 9. Figure 9: Multi-GPU end-to-end decode latency (M=1) on Llama-2-13B and Llama-2-70B at TP=2/3/4. fused TP execution pipeline. 5.3.3 SLO-Compliant Chunk Scheduling We evaluate whether SPEAR preserves a stable latency–throughput tradeoff under continuous batch￾ing as EC selection varies [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Per-token compensation recovery by error [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Per-token cosine similarity between FP16 and 4-bit-quantized hidden states across nine input [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 13
Figure 13. Figure 13: Granularity-driven shift on Llama-2-7B, 4-bit RTN: per-channel (top) vs. group-128 (bottom). C.3 Cross-Quantizer Sensitivity Shift The effect that motivates per-configuration diagnosis is not a change in the rank order of module sensitivity across quantizers, but a change in the membership of the top-K% compensation set: at any operating point, the modules SPEAR actually instruments differ [PITH_FULL_IMA… view at source ↗
Figure 12
Figure 12. Figure 12: Quantizer-driven shift on Llama-2-7B, 3- [PITH_FULL_IMAGE:figures/full_fig_p025_12.png] view at source ↗
read the original abstract

Efficient large language model (LLM) serving is increasingly constrained by deployment cost. Quantization is a key technique for reducing serving cost, yet even state-of-the-art 4-bit quantizers exhibit a noticeable quality gap from FP16, particularly for smaller models where low-bit serving is most beneficial. We identify a fundamental cause of this gap: quantization error is highly input-dependent and varies substantially across tokens, while existing post-quantization compensation methods are static and apply identical corrections to all inputs. As a result, easy tokens are over-corrected while hard tokens remain under-corrected. We present SPEAR, a system for post-quantization error-adaptive recovery that improves low-bit LLM serving. SPEAR introduces lightweight Error Compensators (ECs) modulated by per-token gates and places them only at the most error-sensitive layers identified through a CKA-guided entropy-aware diagnostic. This focuses a small parameter budget where it is most effective. Efficient deployment of ECs presents several systems challenges, including additional computation, tensor-parallel synchronization caused by input-dependent gating, and latency instability across configurations. SPEAR addresses these issues through adaptive kernel-fusion dispatch, combining an epilogue-integrated peer-reduction kernel with P2P dual-write to fuse the post-EC computation into low-bit GEMMs, and an SLO-constrained EC-aware scheduler for predictable serving performance. Across challenging per-channel quantization settings, SPEAR recovers 56-75% of the perplexity gap between W4 and FP16 while adding less than 1% model memory overhead and maintaining latency comparable to a widely used 4-bit serving deployment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript presents SPEAR, a system for post-quantization error-adaptive recovery in low-bit LLM serving. It identifies input-dependent quantization error as the source of the quality gap in 4-bit models and introduces lightweight Error Compensators (ECs) modulated by per-token gates, placed only at layers selected via a CKA-guided entropy-aware diagnostic. Deployment challenges (added compute, tensor-parallel synchronization from input-dependent gating, and latency instability) are addressed via adaptive kernel-fusion dispatch that combines an epilogue-integrated peer-reduction kernel with P2P dual-write to fuse post-EC computation into low-bit GEMMs, plus an SLO-constrained EC-aware scheduler. The central claim is that, across challenging per-channel quantization settings, SPEAR recovers 56-75% of the perplexity gap to FP16 while adding <1% model memory overhead and maintaining latency comparable to a standard 4-bit serving deployment.

Significance. If the empirical claims hold under broader scrutiny, the work could meaningfully improve the quality-efficiency tradeoff for quantized LLM inference by shifting from static to input-adaptive compensation with negligible overhead. The explicit treatment of tensor-parallel synchronization and scheduler integration for per-token mechanisms is a practical strength; the kernel-fusion approach supplies a concrete, implementable path that other systems papers can build upon.

major comments (2)
  1. [Abstract] Abstract: the claim that the described mechanisms 'address tensor-parallel synchronization caused by input-dependent gating' and 'latency instability' while 'maintaining latency comparable' is load-bearing for the efficiency half of the central result, yet the kernel-fusion description supplies no quantitative bound on residual synchronization or dispatch cost when gate decisions differ across ranks or at high tensor-parallel degree.
  2. [Abstract] Abstract / evaluation description: the reported 56-75% recovery figures are presented as direct measurements without error bars, without enumeration of models or datasets, and without an ablation isolating the CKA-guided diagnostic from post-hoc layer selection; this weakens confidence that the gains are robust rather than configuration-specific.
minor comments (1)
  1. The abstract would be clearer if it named the specific quantization method (e.g., per-channel) and the baseline 4-bit serving system used for the latency comparison.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and will revise the abstract for greater precision and completeness.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that the described mechanisms 'address tensor-parallel synchronization caused by input-dependent gating' and 'latency instability' while 'maintaining latency comparable' is load-bearing for the efficiency half of the central result, yet the kernel-fusion description supplies no quantitative bound on residual synchronization or dispatch cost when gate decisions differ across ranks or at high tensor-parallel degree.

    Authors: We agree the abstract would be improved by an explicit quantitative bound. The evaluation section reports latency measurements across TP degrees and gate-variation scenarios; we will revise the abstract to reference these bounds (e.g., residual dispatch overhead remains below the level that affects end-to-end comparability) and clarify the kernel-fusion limits on synchronization cost. revision: yes

  2. Referee: [Abstract] Abstract / evaluation description: the reported 56-75% recovery figures are presented as direct measurements without error bars, without enumeration of models or datasets, and without an ablation isolating the CKA-guided diagnostic from post-hoc layer selection; this weakens confidence that the gains are robust rather than configuration-specific.

    Authors: The 56-75% range aggregates results over the models and datasets enumerated in Section 4; error bars appear in the corresponding figures, and the CKA ablation versus post-hoc selection is shown in Section 5.2. We will revise the abstract to list the models/datasets and explicitly reference the ablation study. revision: partial

Circularity Check

0 steps flagged

No circularity in derivation chain

full rationale

The paper presents its central claims as empirical measurements of perplexity recovery (56-75% gap closure) against an FP16 reference under per-channel quantization, with system overheads reported as direct observations. No equations, fitted parameters renamed as predictions, or self-citation chains are invoked to derive these outcomes by construction. The CKA-guided placement, per-token gating, and kernel fusions are described as engineering choices whose effectiveness is validated externally via benchmarks rather than reduced to inputs. This matches the default case of a self-contained empirical systems paper.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no explicit free parameters, axioms, or invented entities beyond the high-level description of ECs and the diagnostic can be extracted.

pith-pipeline@v0.9.1-grok · 5840 in / 1087 out tokens · 19699 ms · 2026-06-27T23:02:00.077833+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

61 extracted references · 1 canonical work pages

  1. [1]

    Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles , pages=

    Aegaeon: Effective GPU pooling for concurrent LLM serving on the market , author=. Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles , pages=

  2. [2]

    Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 , pages=

    Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-flow , author=. Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 , pages=

  3. [3]

    Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , pages=

    Tapas: Thermal-and Power-aware Scheduling for LLM Inference in Cloud Platforms , author=. Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , pages=

  4. [4]

    Proceedings of the twentieth European conference on computer systems , pages=

    Cacheblend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion , author=. Proceedings of the twentieth European conference on computer systems , pages=

  5. [5]

    Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles , pages=

    JENGA: Effective Memory Management for Serving LLM with Heterogeneity , author=. Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles , pages=

  6. [6]

    Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles , pages=

    Diffkv: Differentiated Memory Management for Large Language Models with Parallel KV Compaction , author=. Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles , pages=

  7. [7]

    Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles , pages=

    Ic-cache: Efficient Large Language Model Serving via In-context Caching , author=. Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles , pages=

  8. [8]

    2025 USENIX Annual Technical Conference (USENIX ATC 25) , pages=

    Weaver: Efficient \ Multi-LLM \ Serving with Attention Offloading , author=. 2025 USENIX Annual Technical Conference (USENIX ATC 25) , pages=

    2025

  9. [9]

    19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25) , pages=

    Mirage: A \ Multi-Level \ Superoptimizer for Tensor Programs , author=. 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25) , pages=

  10. [10]

    Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , pages=

    Pim is all you need: A CXL-enabled GPU-free System for Large Language Model Inference , author=. Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , pages=

  11. [11]

    Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

    Self-instruct: Aligning Language Models with Self-generated Instructions , author=. Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

  12. [12]

    Proceedings of Machine Learning and Systems , volume=

    Qserve: W4a8kv4 Quantization and System Co-design for Efficient LLM Serving , author=. Proceedings of Machine Learning and Systems , volume=

  13. [13]

    Proceedings of the Twentieth European Conference on Computer Systems , pages=

    T-mac: CPU Renaissance via Table Lookup for Low-bit LLM Deployment on Edge , author=. Proceedings of the Twentieth European Conference on Computer Systems , pages=

  14. [14]

    2024 USENIX Annual Technical Conference (USENIX ATC 24) , pages=

    \ Quant-LLM \ : Accelerating the Serving of Large Language Models via \ FP6-Centric \ \ Algorithm-System \ \ Co-Design \ on Modern \ GPUs \ , author=. 2024 USENIX Annual Technical Conference (USENIX ATC 24) , pages=

    2024

  15. [15]

    Advances in neural information processing systems , volume=

    Qlora: Efficient Finetuning of Quantized LLMs , author=. Advances in neural information processing systems , volume=

  16. [16]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Piqa: Reasoning about Physical Commonsense in Natural Language , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  17. [17]

    arXiv preprint arXiv:2009.03300 , year=

    Measuring Massive Multitask Language Understanding , author=. arXiv preprint arXiv:2009.03300 , year=

    Pith/arXiv arXiv 2009

  18. [18]

    arXiv preprint arXiv:1803.05457 , year=

    Think You Have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge , author=. arXiv preprint arXiv:1803.05457 , year=

    Pith/arXiv arXiv

  19. [19]

    Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

    HellaSwag: Can a Machine Really Finish your Sentence? , author=. Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

  20. [20]

    Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    The LAMBADA Dataset: Word Prediction Requiring a Broad Discourse Context , author=. Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

  21. [21]

    BoolQ: Exploring the Surprising Difficulty of Natural yes/no Questions , author=. Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers) , pages=

    2019

  22. [22]

    2026 , publisher =

    Lintang Sutawika and Hailey Schoelkopf and Leo Gao and Baber Abbasi and Stella Biderman and Jonathan Tow and ben fattori and Charles Lovering and farzanehnakhaee70 and Jason Phang and Anish Thite and Fazz and Thomas Wang and Niklas and Aflah and sdtblck and nopperl and gakada and tttyuntian and researcher2 and Julen Etxaniz and Chris and James A. Michaelo...

    work page doi:10.5281/zenodo.18636344

  23. [23]

    Communications of the ACM , volume=

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale , author=. Communications of the ACM , volume=. 2021 , publisher=

    2021

  24. [24]

    Edward J Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen , booktitle=. Lo

  25. [25]

    Advances in neural information processing systems , volume=

    Language Models are Few-shot Learners , author=. Advances in neural information processing systems , volume=

  26. [26]

    arXiv preprint arXiv:2210.17323 , year=

    Gptq: Accurate Post-training Quantization for Generative Pre-trained Transformers , author=. arXiv preprint arXiv:2210.17323 , year=

    Pith/arXiv arXiv

  27. [27]

    Proceedings of machine learning and systems , volume=

    Awq: Activation-aware Weight Quantization for On-device LLM Compression and Acceleration , author=. Proceedings of machine learning and systems , volume=

  28. [28]

    arXiv preprint arXiv:2310.08659 , year=

    Loftq: Lora-fine-tuning-aware Quantization for Large Language Models , author=. arXiv preprint arXiv:2310.08659 , year=

    arXiv

  29. [29]

    arXiv preprint arXiv:2410.21271 , year=

    EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation , author=. arXiv preprint arXiv:2410.21271 , year=

    arXiv

  30. [30]

    International conference on machine learning , pages=

    Similarity of Neural Network Representations Revisited , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  31. [31]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and Fine-tuned Chat Models , author=. arXiv preprint arXiv:2307.09288 , year=

    Pith/arXiv arXiv

  32. [32]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Owq: Outlier-aware Weight Quantization for Efficient Fine-tuning and Inference of Large Language Models , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  33. [33]

    and Keutzer, Kurt , booktitle =

    Kim, Sehoon and Hooper, Coleman Richard Charles and Gholami, Amir and Dong, Zhen and Li, Xiuyu and Shen, Sheng and Mahoney, Michael W. and Keutzer, Kurt , booktitle =. 2024 , editor =

    2024

  34. [34]

    Advances in Neural Information Processing Systems , volume=

    Quarot: Outlier-free 4-bit Inference in Rotated LLMs , author=. Advances in Neural Information Processing Systems , volume=

  35. [35]

    The Thirteenth International Conference on Learning Representations , pages=

    SpinQuant: LLM Quantization with Learned Rotations , author=. The Thirteenth International Conference on Learning Representations , pages=

  36. [36]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  37. [37]

    The Twelfth International Conference on Learning Representations , pages=

    OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models , author=. The Twelfth International Conference on Learning Representations , pages=

  38. [38]

    Advances in neural information processing systems , volume=

    Quip: 2-bit Quantization of Large Language Models with Guarantees , author=. Advances in neural information processing systems , volume=

  39. [39]

    Neural Information Processing Systems , year=

    The Llama 3 herd of models , author=. Neural Information Processing Systems , year=

  40. [40]

    Advances in Neural Information Processing Systems , volume=

    Optimal Brain Compression: A Framework for Accurate Post-training Quantization and Pruning , author=. Advances in Neural Information Processing Systems , volume=

  41. [41]

    Advances in Neural Information Processing Systems , volume=

    Duquant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs , author=. Advances in Neural Information Processing Systems , volume=

  42. [42]

    Advances in neural information processing systems , volume=

    Magr: Weight Magnitude Reduction for Enhancing Post-Training Quantization , author=. Advances in neural information processing systems , volume=

  43. [43]

    The Twelfth International Conference on Learning Representations , pages=

    AffineQuant: Affine Transformation Quantization for Large Language Models , author=. The Twelfth International Conference on Learning Representations , pages=

  44. [44]

    The Thirteenth International Conference on Learning Representations , year =

    OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting , author=. The Thirteenth International Conference on Learning Representations , year =

  45. [45]

    2025 , volume =

    Sun, Yuxuan and Liu, Ruikang and Bai, Haoli and Bao, Han and Zhao, Kang and Li, Yuening and Hu, Jiaxin and Yu, Xianzhi and Hou, Lu and Yuan, Chun and Jiang, Xin and Liu, Wulong and Yao, Jun , booktitle =. 2025 , volume =

    2025

  46. [46]

    in Post-Training Quantization , author =

    Qronos: Correcting the Past by Shaping the Future... in Post-Training Quantization , author =. The Fourteenth International Conference on Learning Representations , year =

  47. [47]

    and Zhao, Yiren , title =

    Zhang, Cheng and Cheng, Jianyi and Constantinides, George A. and Zhao, Yiren , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , publisher =

    2024

  48. [48]

    The Thirteenth International Conference on Learning Representations , year =

    QERA: an Analytical Framework for Quantization Error Reconstruction , author=. The Thirteenth International Conference on Learning Representations , year =

  49. [49]

    2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages=

    HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision , author=. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages=. 2019 , organization=

    2019

  50. [50]

    arXiv preprint arXiv:2505.22988 , year=

    Model-preserving Adaptive Rounding , author=. arXiv preprint arXiv:2505.22988 , year=

    Pith/arXiv arXiv

  51. [51]

    Forty-second International Conference on Machine Learning , year =

    GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration , author=. Forty-second International Conference on Machine Learning , year =

  52. [52]

    2023 , editor =

    Xiao, Guangxuan and Lin, Ji and Seznec, Mickael and Wu, Hao and Demouth, Julien and Han, Song , booktitle =. 2023 , editor =

    2023

  53. [53]

    2024 , editor =

    Tseng, Albert and Chee, Jerry and Sun, Qingyao and Kuleshov, Volodymyr and De Sa, Christopher , booktitle =. 2024 , editor =

    2024

  54. [54]

    Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming , pages=

    Marlin: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models , author=. Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming , pages=

  55. [55]

    Proceedings of the 29th symposium on operating systems principles , pages=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the 29th symposium on operating systems principles , pages=

  56. [56]

    18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , year =

    Amey Agrawal and Nitin Kedia and Ashish Panwar and Jayashree Mohan and Nipun Kwatra and Bhargav Gulavani and Alexey Tumanov and Ramachandran Ramjee , title =. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , year =

  57. [57]

    18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages=

    \ DistServe \ : Disaggregating Prefill and Decoding for Goodput-optimized Large Language model Serving , author=. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages=

  58. [58]

    Advances in neural information processing systems , volume=

    Flashattention: Fast and Memory-efficient Exact Attention with IO-awareness , author=. Advances in neural information processing systems , volume=

  59. [59]

    Proceedings of Machine Learning and Systems , volume=

    Atom: Low-bit Quantization for Efficient and Accurate LLM Serving , author=. Proceedings of Machine Learning and Systems , volume=

  60. [60]

    arXiv preprint arXiv:2311.03285 , year=

    S-LoRA: Serving Thousands of Concurrent LoRA Adapters , author=. arXiv preprint arXiv:2311.03285 , year=

    arXiv

  61. [61]

    Advances in Neural Information Processing Systems , volume=

    Flashattention-3: Fast and Accurate Attention with Asynchrony and Low-precision , author=. Advances in Neural Information Processing Systems , volume=