Pith. sign in

REVIEW 4 major objections 6 minor 60 references

Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Storing LLM weights in 4-bit integers while computing in FP8 keeps accuracy near BF16 and promises up to 3x speedup over W4A16, via dual-precision error compensation and group-aware reordering.

desk verdict Solid accuracy story for a W4A8 scheme with a genuinely useful reordering trick; the speedup numbers are projections, not measurements, and should be read as the paper's main caveat. read the letter →

arxiv 2505.14638 v1 pith:TOALUIUF submitted 2025-05-20 cs.CV cs.CLcs.LG

Tomer Gafni , Asaf Karnieli , Yair Hanani This is my paper
classification cs.CVcs.CLcs.LG
keywords post-trainingquantizationW4A8FP8inferenceINT4weightstorageDualPrecisionGroup-AwareReorderinglargelanguagemodelsvision-language
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a hybrid W4A8 quantization scheme — weights stored in 4-bit integers, all matrix multiplications computed in 8-bit floating point — can run large language and vision-language models at near-BF16 accuracy while cutting memory traffic and exploiting FP8 tensor cores. The authors argue that prior W4A8 schemes quantize activations to INT8, whose uniform grid fits neither the roughly normal distribution of weights nor the outlier-heavy distribution of activations, whereas FP8's non-uniform grid and dynamic range preserve more accuracy. To make the double quantization (INT4 storage plus FP8 arithmetic) accurate, they introduce Dual Precision Quantization (DPQ), which compensates quantization error through both levels using Hessian-based updates, and Group-Aware Reordering (GAR), which prioritizes important weights without breaking the group layout that keeps inference overhead low. If the scheme works as claimed, models can be stored at half the memory of FP8 weights and computed at FP8 speed with accuracy close to the full-precision baseline.

What carries the argument

The load-bearing object is the two-step dequantization chain inside DPQ: a weight is quantized from BF16 to FP8, then from FP8 to INT4 for storage; during inference it is dequantized INT4 → FP8 → BF16. The error term fed into the Hessian update is computed from the final BF16 reconstruction, so both quantization levels are compensated simultaneously rather than separately. The second mechanism, Group-Aware Reordering (GAR), permutes weights by Hessian diagonal importance subject to the constraint that permutations happen only inside groups or by swapping whole groups, which guarantees that after the tensor is re-permuted back to its original order each group's scale and zero-point still apply to consecutive weights; this preserves the single-multiplication dequantization of the original scheme. Hessian dampening and Cholesky-based computation of the inverse Hessian make the updates tractable, and power-of-two FP8 scales let the hardware fold scaling into the exponent bias.

What would settle it

Measure end-to-end tokens per second on Gaudi 2 or Gaudi 3 for Llama-2 70B and Qwen2-72B quantized with DPQ at W4A8, and compare against W4A16 and W8A8 at matched batch sizes and sequence lengths; if the measured speedup does not reach the projected curves in Fig. 4, the performance claim is refuted. A second check would compare DPQ's FP8 activations against an INT8-activation W4A8 scheme with GAR applied to both, isolating whether the accuracy gain comes from the FP8 grid or from the reordering.

Watch

Extended reading notes

Core claim

The central claim is that a W4A8 scheme with FP8 activations and computations, rather than INT8, is both more accurate and equally hardware-friendly, provided the weight quantization error is compensated across the full INT4-to-FP8-to-BF16 dequantization chain. DPQ does this by quantizing one weight at a time, dequantizing the stored INT4 value back through FP8 to BF16, and distributing the resulting two-level error to the remaining weights scaled by the inverse Hessian, exactly the second-order error-compensation update used in layer-wise quantization. GAR constrains the reordering to permutations within groups and whole-group swaps, so the per-group scales and zero-points stay aligned with consecutive memory after the tensor is restored to its original order; this removes the indexing overhead of unrestricted reordering while still quantizing the most Hessian-important weights first. The paper reports that on Qwen2-VL, Llama-2, and Llama-3, DPQ keeps accuracy within a few percent of BF16 on vision-language benchmarks, perplexity, common-sense reasoning, and MMLU, and outperforms existing W4A8 schemes that use INT8 activations. Performance projections on Gaudi 2 and 3 show up to 3x speedup over W4A16 and up to 1.4x over W8A8.

Load-bearing premise

The headline speedups come from LLM model projections rather than measured end-to-end throughput, so the load-bearing premise is that real Gaudi 2 and 3 runs will deliver the assumed FP8 matrix-multiplication utilization and memory bandwidth; if those assumptions are optimistic, the claimed gains shrink or vanish.

Editorial extensions

If this is right

  • A W4A8 model quantized with DPQ and GAR stores weights in INT4, halving the memory footprint relative to W8A8 and cutting the bandwidth needed in memory-bound autoregressive decoding.
  • Because all matrix multiplications run in FP8, the scheme can use the doubled FP8 throughput of modern accelerators, so the same model serves larger batches than a W4A16 deployment.
  • The accuracy gap to the BF16 baseline is small enough on 7B-70B language models and 2B-72B vision-language models that W4A8 can replace W4A16 in serving settings where latency and memory dominate.
  • Since GAR keeps scales aligned with consecutive weights, it can be dropped into other group-wise quantization schemes such as W4A16 without adding inference overhead.
  • The reported speedups of up to 3x over W4A16 and 1.4x over W8A8 follow from combining the smaller INT4 memory footprint with FP8 compute, and they grow with input length in the projections.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's own results, the ablation suggests GAR's constrained reordering, not the FP8 grid, is the main accuracy lever, so a clean comparison of FP8 versus INT8 activations should hold the reordering scheme fixed.
  • The same two-level error compensation could be applied to other mixed-precision settings, such as INT4 storage with FP8 weight-only dequantization or KV-cache quantization, without changing inference cost.
  • The paper's speedup curves are projections, so a direct measurement of end-to-end tokens per second on Gaudi 2 and 3 across batch sizes would test the assumed FP8 utilization and memory bandwidth.
  • Because FP8's advantage over INT8 is largest for outlier-heavy activation distributions, models with known massive activation outliers should show a larger accuracy gap between the two formats; the paper's own discussion points to this test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a W4A8 post-training quantization scheme for LLM/VLM inference: weights are stored as asymmetric INT4 with per-group scales and zero-points, then dequantized to FP8 at runtime, while activations are quantized to FP8 for FP8 matrix multiplication. To control accuracy loss from the two-stage weight quantization, the authors introduce DPQ, a GPTQ/OBQ-style algorithm that compensates quantization errors using Hessian information computed on the dequantized (INT4-to-FP8-to-BF16) weights, and GAR, a constrained reordering scheme that permutes weights within groups and permutes whole groups so that group scales remain aligned with consecutive memory locations at inference. Accuracy is evaluated on Qwen2-VL (MMMU, MMBench, MathVista) and Llama-2/Llama-3 (WikiText-2, common-sense reasoning, MMLU), with comparisons to QServe, QQQ, GPTQ, and RTN. Throughput is presented as LLM model projections in Fig. 4, claiming up to 3x speedup over W4A16 and up to 1.4x speedup over W8A8 on Gaudi 2/3.

Significance. If the accuracy results are representative, the paper makes a useful contribution: the W4A8 format with FP8 compute and INT4 storage is relevant to both memory-bound and compute-bound inference regimes, and GAR appears applicable beyond this specific setting. The accuracy experiments use independent external baselines (QServe, QQQ) and show that DPQ is competitive with or better than these methods on most reported tasks, which is a genuine strength; the code is also released through Intel Neural Compressor. However, the paper's central efficiency claim currently rests on unmeasured projections with unspecified assumptions, and the power-of-two scaling claim is not reconciled with the quantization equations, so the main practical advantage is not yet established.

major comments (4)
  1. [Sec. 4.3, Fig. 4] The headline speedups ('up to 3x speed-up over W4A16, and up to 1.4x speed-up over W8A8') are based entirely on 'LLM model projections' rather than measured end-to-end throughput. The text never specifies the projection model: no assumed memory bandwidth, FP8 GeMM utilization, batch-size scaling, or cost of the online INT4-to-FP8 dequantization shown in Fig. 3 (read INT4, subtract zero-point, multiply by per-group scale and FP8 scale). Since this dequantization is an extra step relative to W8A8, the 1.4x claim over W8A8 cannot be assessed without knowing whether and how it is modeled. Please report measured Gaudi 2/3 throughput or fully specify the projection methodology, including assumptions and a sensitivity analysis.
  2. [Sec. 2.2, Eqs. (6), (8), (13), Sec. 3.3] The paper says power-of-two FP8 scales are incorporated to exploit Gaudi's exponent-bias optimization, but the algorithm as defined does not produce power-of-two scales: Eq. (6) sets sF = max(|W|)/Zmax, Eq. (13) does the same for activations, Eq. (8) computes sg from min/max, and Sec. 3.3 states the group scale is found by MSE search. These are generally not powers of two. Please state explicitly how (or whether) scales are rounded or constrained to powers of two and whether the Fig. 4 projections assume the exponent-bias optimization; otherwise the claimed hardware advantage is not connected to the quantization procedure.
  3. [Table 3 vs. Table 4] The full-reordering W4A8 results are inconsistent between tables. For Llama-3.1-8B, Table 3 reports W4A8* (DPQ), described as using full activation reordering, with WikiText-2 perplexity 6.935, while Table 4 reports the 'Ordered' W4A8 configuration with perplexity 6.679 under the same BF16 reference (6.238). These two entries should describe the same configuration; please reconcile them and clarify which result supports the claim that GAR is only slightly worse than full reordering.
  4. [Sec. 3.2, Fig. 3] The paper repeatedly states that DPQ/GAR achieves accuracy 'without introducing any inference overhead,' but the W4A8 flow in Fig. 3 includes an online step that reads INT4 weights, subtracts the zero-point, multiplies by the per-group scale, and then applies FP8 scaling. GAR removes the indexing overhead of full reordering, but it does not remove this arithmetic. Please either report the measured cost of this dequantization in the throughput numbers or qualify the 'no overhead' claim to mean 'no additional indexing overhead relative to no-reorder group quantization.'
minor comments (6)
  1. [Sec. 4.4, Appendix A] The text uses 'activation ordering' and 'activation reordering' to describe operations applied to weight groups; this terminology is likely to confuse readers, since the tensor being permuted is the weight tensor. Please use 'weight reordering guided by activation/Hessian statistics' consistently.
  2. [Table 2] The column headers 'A' and 'VG' are not defined; if 'A' is the average over the common-sense suite, say so and list all component tasks.
  3. [Sec. 3.1, Eq. (12)] The notation w16 and \widehat{w}16 is introduced in the text but the equation uses both; please use a consistent symbol and define \widehat{w}16 as the two-step dequantized value before Eqs. (10)-(11).
  4. [Table 3, footnote] The sentence 'since DPQ adds an additional level of quantization over GPTQ, it serves as an upper-bound to DPQ' is unclear; presumably W4A16 GPTQ is an upper bound for W4A8 DPQ, but the phrasing should be corrected.
  5. [Fig. 4 caption] Each subplot's legend should state explicitly that all curves are normalized relative to W4A16, and the input-token ranges and the assumed output length should be given in the caption.
  6. [Sec. 4.3] The statement that Llama 3.2 90B Vision can fit in an Nvidia H100 (80 GB HBM) with 4-bit weights should include the KV-cache and activation memory budget used for the estimate; as written, a 90B-parameter model at 4-bit weights plus activations may not fit an 80 GB device in all serving configurations.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: DPQ accuracy is evaluated against independent benchmarks and baselines, and the W4A8 throughput projections, while under-specified, are not shown to reduce to the paper's own inputs.

full rationale

The accuracy chain is self-contained: DPQ minimizes the layer-wise squared error in Eq. (5) with the two-step dequantization defined in Eqs. (6)-(12), the calibration set is disjoint from the evaluation benchmarks, and no accuracy target is used to set a fitted constant. Comparisons against QServe, QQQ, GPTQ, and RTN use external baselines (Tables 2-3), and the GAR ablation is a controlled comparison, not a prediction derived from the method's own success criterion. The throughput claim in Sec. 4.3 is explicitly presented as an 'LLM model projection' (Fig. 4) whose projection model is not stated, and the power-of-two optimization is cited to the authors' own prior work [19]; however, no equation or fitted value in this paper is exhibited as equivalent to the projected speedups, so this is an evidence-transparency and correctness risk rather than a demonstrated circular reduction. The self-citation of [19] is present but not load-bearing in a circular sense: it points to a hardware behavior that is externally falsifiable on Gaudi devices, and the central algorithmic accuracy claim does not depend on it.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central accuracy claim does not depend on exotic invented quantities; it depends on standard GPTQ machinery plus the group size and dampening hyperparameters, and on the assumption that calibration activations represent the real input distribution. No new physical entities are introduced.

free parameters (2)
  • INT4 group size = 128
    Used for all experiments; controls the granularity of per-group scales and zero-points. No sensitivity analysis is reported, so its role in the accuracy results is not isolated.
  • Hessian dampening factor = not reported
    Paper states 'we apply dampening to the Hessian matrix' following GPTQ, but never gives the value or schedule used in the experiments. This affects every weight update in Eq. (12).
assumptions (3)
  • standard math OBQ inverse-Hessian update formula (Eq. 12) gives the optimal error compensation for each quantized weight.
    The derivation is cited to [36] and not re-derived; DPQ's central error-compensation step inherits its optimality assumptions.
  • domain assumption The Hessian H = X^T X computed from calibration activations approximates the true layer-wise loss curvature.
    Standard GPTQ approximation; assumes calibration data represents deployment distribution and that the layer-wise squared error is the right objective. Enters in Sec. 3.1 and Appendix A.2.
  • ad hoc to paper Ranking weight groups by the maximum diagonal Hessian element is an adequate importance ordering.
    The paper explicitly says alternative ranking criteria are possible and leaves exploration to future work (Sec. 4.4, Appendix A.3), so this ranking is a design choice, not a theorem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference." pith.science (2026). https://pith.science/paper/TOALUIUF

@misc{pith2026250514638,
  author       = {Pith},
  title        = {Pith review of: Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TOALUIUF}},
  note         = {Machine review of arXiv:2505.14638}
}
read the original abstract

Deep neural networks have achieved state-of-the-art results in a wide range of applications, from natural language processing and computer vision to speech recognition. However, as tasks become increasingly complex, model sizes continue to grow, posing challenges in latency and memory efficiency. To meet these constraints, post-training quantization has emerged as a promising solution. In this paper, we propose a novel hardware-efficient quantization and inference scheme that exploits hardware advantages with minimal accuracy degradation. Specifically, we introduce a W4A8 scheme, where weights are quantized and stored using 4-bit integer precision, and inference computations are performed using 8-bit floating-point arithmetic, demonstrating significant speedups and improved memory utilization compared to 16-bit operations, applicable on various modern accelerators. To mitigate accuracy loss, we develop a novel quantization algorithm, dubbed Dual Precision Quantization (DPQ), that leverages the unique structure of our scheme without introducing additional inference overhead. Experimental results demonstrate improved performance (i.e., increased throughput) while maintaining tolerable accuracy degradation relative to the full-precision model.

Figures

Figures reproduced from arXiv: 2505.14638 by the authors.

Figure 1
Figure 1. An illustration of the GAR method. In this example, [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. An illustration of the inference flow of our proposed [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Speed-up comparison between 3 methods on Gaudi 2 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: An illustration of per-group scale and zero-point. In [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Computing scales and zero-points under full activation [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Computing scales and zero-points under GAR method. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 32 canonical work pages

  1. [1]

    Learned step size quantization,

    S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha, “Learned step size quantization,” arXiv preprint arXiv:1902.08153, 2019. 1

    arXiv 1902

  2. [2]

    Ef- fective training of convolutional neural networks with low- bitwidth weights and activations,

    B. Zhuang, M. Tan, J. Liu, L. Liu, I. Reid, and C. Shen, “Ef- fective training of convolutional neural networks with low- bitwidth weights and activations,”IEEE Transactions on Pat- tern Analysis and Machine Intelligence , vol. 44, no. 10, pp. 6140–6152, 2021

    work page 2021

  3. [3]

    Cluster- promoting quantization with bit-drop for minimizing net- work quantization loss,

    J. H. Lee, J. Yun, S. J. Hwang, and E. Yang, “Cluster- promoting quantization with bit-drop for minimizing net- work quantization loss,” inProceedings of the IEEE/CVF In- ternational Conference on Computer Vision, 2021, pp. 5370– 5379

    work page 2021

  4. [4]

    Llm-qat: Data- free quantization aware training for large language models,

    Z. Liu, B. Oguz, C. Zhao, E. Chang, P. Stock, Y . Mehdad, Y . Shi, R. Krishnamoorthi, and V . Chandra, “Llm-qat: Data- free quantization aware training for large language models,” arXiv preprint arXiv:2305.17888, 2023. 1

    arXiv 2023

  5. [5]

    Data-free quantization through weight equalization and bias correction,

    M. Nagel, M. v. Baalen, T. Blankevoort, and M. Welling, “Data-free quantization through weight equalization and bias correction,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1325–1334. 1

    work page 2019

  6. [6]

    Improving post training neural quantization: Layer-wise calibration and integer programming,

    I. Hubara, Y . Nahshan, Y . Hanani, R. Banner, and D. Soudry, “Improving post training neural quantization: Layer-wise calibration and integer programming,” arXiv preprint arXiv:2006.10518, 2020

    arXiv 2006

  7. [7]

    Accurate post training quantization with small calibra- tion sets,

    I. Hubara, Y . Nahshan, Y . Hanani, R. Banner, and D.Soudry, “Accurate post training quantization with small calibra- tion sets,” in International conference on machine learning. PMLR, 2021, pp. 4466–4475

    work page 2021

  8. [8]

    Post-training quantization for vision transformer,

    Z. Liu, Y . Wang, K. Han, W. Zhang, S. Ma, and W. Gao, “Post-training quantization for vision transformer,” Ad- vances in Neural Information Processing Systems , vol. 34, pp. 28 092–28 103, 2021. 2

    work page 2021

Show all 60 references
  1. [9]

    Loss aware post-training quantization,

    Y . Nahshan, B. Chmiel, C. Baskin, E. Zheltonozhskii, R. Banner, A. M. Bronstein, and A. Mendelson, “Loss aware post-training quantization,” Machine Learning , vol. 110, no. 11, pp. 3245–3262, 2021

    2021

  2. [10]

    Zeroquant: Efficient and affordable post-training quantization for large-scale transformers,

    Z. Yao, R. Yazdani Aminabadi, M. Zhang, X. Wu, C. Li, and Y . He, “Zeroquant: Efficient and affordable post-training quantization for large-scale transformers,” Advances in Neu- ral Information Processing Systems , vol. 35, pp. 27 168– 27 183, 2022. 1

    2022

  3. [11]

    Up or down? adaptive rounding for post- training quantization,

    M. Nagel, R. A. Amjad, M. Van Baalen, C. Louizos, and T. Blankevoort, “Up or down? adaptive rounding for post- training quantization,” in International conference on ma- chine learning. PMLR, 2020, pp. 7197–7206. 1, 2

    2020

  4. [12]

    Gptq: Accurate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323,

    arXiv

  5. [13]

    Awq: Activation- aware weight quantization for on-device llm compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “Awq: Activation- aware weight quantization for on-device llm compression and acceleration,” Proceedings of Machine Learning and Systems, vol. 6, pp. 87–100, 2024. 1, 2, 3

    2024

  6. [14]

    Q-vlm: Post-training quantization for large vision-language models,

    C. Wang, Z. Wang, X. Xu, Y . Tang, J. Zhou, and J. Lu, “Q-vlm: Post-training quantization for large vision-language models,” Advances in Neural Information Processing Sys- tems, vol. 37, pp. 114 553–114 573, 2025. 1

    2025

  7. [15]

    Advancing multimodal large language models with quantization-aware scale learning for efficient adaptation,

    J. Xie, Y . Zhang, M. Lin, L. Cao, and R. Ji, “Advancing multimodal large language models with quantization-aware scale learning for efficient adaptation,” inProceedings of the 32nd ACM International Conference on Multimedia , 2024, pp. 10 582–10 591

    2024

  8. [16]

    Reg-ptq: Regression-specialized post-training quantization for fully quantized object detector,

    Y . Ding, W. Feng, C. Chen, J. Guo, and X. Liu, “Reg-ptq: Regression-specialized post-training quantization for fully quantized object detector,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 16 174–16 184

    2024

  9. [17]

    Ptq4sam: Post- training quantization for segment anything,

    C. Lv, H. Chen, J. Guo, Y . Ding, and X. Liu, “Ptq4sam: Post- training quantization for segment anything,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, 2024, pp. 15 941–15 951. 1

    2024

  10. [18]

    Fp8 formats for deep learning,

    P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, S. Ha, A. Heinecke, P. Judd, J. Ka- malu et al., “Fp8 formats for deep learning,” arXiv preprint arXiv:2209.05433, 2022. 1, 2

    2022 arXiv

  11. [19]

    Faster inference of llms using fp8 on the intel gaudi,

    J. Lee, S. Markovich-Golan, D. Ohayon, Y . Hanani, G. Park, B. Kim, A. Karnieli, U. Livne, H. Shen, T. Huang et al. , “Faster inference of llms using fp8 on the intel gaudi,”arXiv preprint arXiv:2503.09975, 2025. 1, 3

    2025 arXiv

  12. [20]

    Fp8 quantization: The power of the expo- nent,

    A. Kuzmin, M. Van Baalen, Y . Ren, M. Nagel, J. Peters, and T. Blankevoort, “Fp8 quantization: The power of the expo- nent,” Advances in Neural Information Processing Systems , vol. 35, pp. 14 651–14 662, 2022

    2022

  13. [21]

    An investigation of fp8 across accelerators for llm inference,

    J. Kim, J. Lee, G. Park, B. Kim, S. J. Kwon, D. Lee, and Y . Lee, “An investigation of fp8 across accelerators for llm inference,” arXiv preprint arXiv:2502.01070, 2025. 2

    2025 arXiv

  14. [22]

    Fp8 versus int8 for efficient deep learning inference,

    M. Van Baalen, A. Kuzmin, S. S. Nair, Y . Ren, E. Mahurin, C. Patel, S. Subramanian, S. Lee, M. Nagel, J. Soriagaet al., “Fp8 versus int8 for efficient deep learning inference,”arXiv preprint arXiv:2303.17951, 2023. 2

    2023 arXiv

  15. [23]

    Qserve: W4a8kv4 quantization and sys- tem co-design for efficient llm serving,

    Y . Lin, H. Tang, S. Yang, Z. Zhang, G. Xiao, C. Gan, and S. Han, “Qserve: W4a8kv4 quantization and sys- tem co-design for efficient llm serving,” arXiv preprint arXiv:2405.04532, 2024. 2, 6, 7

    2024 arXiv

  16. [24]

    The super weight in large language models,

    M. Yu, D. Wang, Q. Shan, and A. Wan, “The super weight in large language models,”arXiv preprint arXiv:2411.07191,

    arXiv

  17. [25]

    Smoothquant: Accurate and efficient post-training quan- tization for large language models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quan- tization for large language models,” in International Con- ference on Machine Learning . PMLR, 2023, pp. 38 087– 38 099. 2

    2023

  18. [26]

    Quarot: Outlier-free 4-bit inference in rotated llms,

    S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman, “Quarot: Outlier-free 4-bit inference in rotated llms,” arXiv preprint arXiv:2404.00456, 2024. 2, 3

    2024 arXiv

  19. [27]

    Squeezellm: Dense- 9 and-sparse quantization,

    S. Kim, C. Hooper, A. Gholami, Z. Dong, X. Li, S. Shen, M. W. Mahoney, and K. Keutzer, “Squeezellm: Dense- 9 and-sparse quantization,” arXiv preprint arXiv:2306.07629,

    arXiv

  20. [28]

    Towards accurate post-training quantization for vi- sion transformer,

    Y . Ding, H. Qin, Q. Yan, Z. Chai, J. Liu, X. Wei, and X. Liu, “Towards accurate post-training quantization for vi- sion transformer,” in Proceedings of the 30th ACM interna- tional conference on multimedia, 2022, pp. 5380–5388. 2

    2022

  21. [29]

    Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers,

    Y . Liu, H. Yang, Z. Dong, K. Keutzer, L. Du, and S. Zhang, “Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 20 321–20 330. 2

    2023

  22. [30]

    Optimize weight rounding via signed gradi- ent descent for the quantization of llms,

    W. Cheng, W. Zhang, H. Shen, Y . Cai, X. He, K. Lv, and Y . Liu, “Optimize weight rounding via signed gradi- ent descent for the quantization of llms,” arXiv preprint arXiv:2309.05516, 2023. 2

    2023 arXiv

  23. [31]

    Omniquant: Omni- directionally calibrated quantization for large language mod- els,

    W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y . Qiao, and P. Luo, “Omniquant: Omni- directionally calibrated quantization for large language mod- els,” arXiv preprint arXiv:2308.13137, 2023

    2023 arXiv

  24. [32]

    Half-quadratic quantization of large machine learning models,

    H. Badri and A. Shaji, “Half-quadratic quantization of large machine learning models,” Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob, 2023

    2023

  25. [33]

    Pd-quant: Post-training quantization based on prediction difference metric,

    J. Liu, L. Niu, Z. Yuan, D. Yang, X. Wang, and L.Wenyu, “Pd-quant: Post-training quantization based on prediction difference metric,” in Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2023, pp. 24 427–24 437

    2023

  26. [34]

    Pd-quant: Post-training quantization based on prediction difference metric,

    J. Liu, L. Niu, Z. Yuan, D. Yang, X. Wang, and W. Liu, “Pd-quant: Post-training quantization based on prediction difference metric,” in Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2023, pp. 24 427–24 437. 2

    2023

  27. [35]

    Optimal brain sur- geon and general network pruning,

    B. Hassibi, D. G. Stork, and G. J. Wolff, “Optimal brain sur- geon and general network pruning,” in IEEE international conference on neural networks. IEEE, 1993, pp. 293–299. 2

    1993

  28. [36]

    Optimal brain compression: A framework for accurate post-training quantization and prun- ing,

    E. Frantar and D. Alistarh, “Optimal brain compression: A framework for accurate post-training quantization and prun- ing,” Advances in Neural Information Processing Systems , vol. 35, pp. 4475–4488, 2022. 4

    2022

  29. [37]

    Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization,

    Z. Yuan, C. Xue, Y . Chen, Q. Wu, and G. Sun, “Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization,” in European conference on computer vision. Springer, 2022, pp. 191–207. 2

    2022

  30. [38]

    Qqq: Quality quattuor- bit quantization for large language models,

    Y . Zhang, P. Zhang, M. Huang, J. Xiang, Y . Wang, C. Wang, Y . Zhang, L. Yu, C. Liu, and W. Lin, “Qqq: Quality quattuor- bit quantization for large language models,” arXiv preprint arXiv:2406.09904, 2024. 2, 6, 7

    2024 arXiv

  31. [39]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” Advances in neural information processing systems, vol. 36, pp. 10 088– 10 115, 2023. 2

    2023

  32. [40]

    Spinquant: Llm quantization with learned rotations,

    Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Kr- ishnamoorthi, V . Chandra, Y . Tian, and T. Blankevoort, “Spinquant: Llm quantization with learned rotations,” arXiv preprint arXiv:2405.16406, 2024. 3

    2024 arXiv

  33. [41]

    Mas- sive activations in large language models,

    M. Sun, X. Chen, J. Z. Kolter, and Z. Liu, “Mas- sive activations in large language models,” arXiv preprint arXiv:2402.17762, 2024. 3, 7, 13

    2024 arXiv

  34. [42]

    Mmmu: A massive multi-discipline multimodal understanding and rea- soning benchmark for expert agi,

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sun et al. , “Mmmu: A massive multi-discipline multimodal understanding and rea- soning benchmark for expert agi,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2024

  35. [43]

    Mmbench: Is your multi-modal model an all-around player?

    Y . Liu, H. Duan, Y . Zhang, B. Li, S. Zhang, W. Zhao, Y . Yuan, J. Wang, C. He, Z. Liu et al., “Mmbench: Is your multi-modal model an all-around player?” in European con- ference on computer vision. Springer, 2024, pp. 216–233. 5

    2024

  36. [44]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts,

    P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao, “Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts,” arXiv preprint arXiv:2310.02255, 2023. 5

    arXiv

  37. [45]

    Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Geet al., “Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution,” arXiv preprint arXiv:2409.12191, 2024. 5

    2024 arXiv

  38. [46]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models,

    H. Duan, J. Yang, Y . Qiao, X. Fang, L. Chen, Y . Liu, X. Dong, Y . Zang, P. Zhang, J. Wang et al. , “Vlmevalkit: An open-source toolkit for evaluating large multi-modality models,” in Proceedings of the 32nd ACM international con- ference on multimedia, 2024, pp. 11 198–11 201. 5

    2024

  39. [47]

    Llama 2: Open foundation and fine-tuned chat mod- els,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat mod- els,” arXiv preprint arXiv:2307.09288, 2023. 5

    2023 arXiv

  40. [48]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Ka- dian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. 5

    2024 arXiv

  41. [49]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843,

    arXiv

  42. [50]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300,

    2009 arXiv

  43. [51]

    Hellaswag: Can a machine really finish your sentence?

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hellaswag: Can a machine really finish your sentence?” arXiv preprint arXiv:1905.07830, 2019. 6

    1905 arXiv

  44. [52]

    Language models are unsupervised mul- titask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised mul- titask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019. 6

    2019

  45. [53]

    Boolq: Exploring the surprising difficulty of natural yes/no questions,

    C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,” arXiv preprint arXiv:1905.10044, 2019. 6

    1905 arXiv

  46. [54]

    Think you have solved ques- tion answering? try arc, the ai2 reasoning challenge,

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved ques- tion answering? try arc, the ai2 reasoning challenge,” arXiv preprint arXiv:1803.05457, 2018. 6 10

    2018 arXiv

  47. [55]

    Reason- ing about physical commonsense in natural language,

    Y . Bisk, R. Zellers, R. Le Bras, J. Gao, and Y . Choi, “Reason- ing about physical commonsense in natural language,” 2019. 6

    2019

  48. [56]

    An adversarial winograd schema challenge at scale,

    K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y . Choi, “An adversarial winograd schema challenge at scale,” arXiv preprint arXiv:1907.10641, 2019. 6

    1907 arXiv

  49. [57]

    Can a suit of armor conduct electricity? a new dataset for open book question answering,

    T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? a new dataset for open book question answering,”arXiv preprint arXiv:1809.02789,

    arXiv

  50. [58]

    A framework for few-shot language model evaluation,

    L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, “A framework...

    2023

  51. [59]

    Semantic parsing on Freebase from question-answer pairs,

    J. Berant, A. Chou, R. Frostig, and P. Liang, “Semantic parsing on Freebase from question-answer pairs,” in Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing . Seattle, Washington, USA: Association for Computational Linguistics, Oct. 2013, ...

    2013

  52. [60]

    The pile: An 800gb dataset of diverse text for language modeling,

    L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, “The pile: An 800gb dataset of diverse text for language modeling,” 2020. [Online]. Available: https://arxiv.org/abs/2101.00027 6 11 Appendix A. G...

    2020 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.