Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Nanoscaling Floating-Point (NxFP): NanoMantissa, Adaptive Microexponents, and Code Recycling for Direct-Cast Compression of Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read NxFP, a new Nanoscaling format, beats MxFP on LLM perplexity while using up to 16 percent less memory.

desk verdict Solid, incremental contribution to low-bit LLM quantization: the NxFP format combination is new, the ablations are honest, and the main soft spot is a tuned constant that needs a transfer test. read the letter →

arxiv 2412.19821 v1 pith:LIXEWA2S submitted 2024-12-15 cs.AR cs.AIcs.DCcs.LG

Yun-Chen Lo , Gu-Yeon Wei , David Brooks This is my paper · ORCID
classification cs.ARcs.AIcs.DCcs.LG
keywords NanoscalingFloating-PointMicroscaling(Mx)low-bitquantizationdirect-castinferencelargelanguagemodelsperplexityMMLUcoderecycling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Nanoscaling Floating-Point (NxFP), a number format for compressing large language models by direct-cast quantization. NxFP augments the Microscaling (MxFP) standard with three techniques: a shared 2-bit NanoMantissa that tracks outlier magnitudes per block, an Adaptive Microexponent bit that lets each block choose between MxFP and block-floating-point layout, and Code Recycling that maps the wasted -0 code to half the smallest quantization step. The paper claims NxFP beats MxFP by up to 0.64 in Wikitext-2 perplexity and up to 30% in MMLU accuracy, and that a 5-bit NxFP model matches 6-bit MxFP perplexity with 13-16% less memory footprint. If correct, this gives sub-6-bit Microscaling formats a path to be usable for generative inference.

What carries the argument

The central object is the NxFP vector format, five fields per block of 32 elements: sign, element mantissa, shared exponent, microexponent (or its absence), and a format bit. The NanoMantissa is a shared 2-bit mantissa stored alongside the shared exponent that scales the whole block by a value $1.m$, letting the format represent values above the largest MxFP code; the Adaptive Microexponent is a single index bit that decides, per block, between the MxFP layout (good for scattered distributions) and the BFP layout (good for clustered distributions); Code Recycling remaps the sign-magnitude code $10...0$, which would be $-0$, to half of the smallest quantization step, recoverable in hardware by a one-bit right shift. The quantization algorithm exercises each choice - NanoMantissa on or off and MxFP or BFP - and picks the combination minimizing MSE, so the three techniques act as a small per-block search over formats. These mechanisms add no bits to the element stream; they only redistribute existing storage.

What would settle it

Quantize Gemma2-2B and Mistral-7B at 4 bits with NxFP while sweeping the recycled code value over the grid {0, 0.25, 0.5, 0.75, 1} times the smallest quantization step, and compare Wikitext-2 perplexity with the fixed half-step choice; if a value other than 0.5 yields meaningfully lower perplexity on these models, the single-constant design is not generally optimal. A second check is to reproduce the reported 4-bit Llama3-8B perplexities of 6.57 for NxFP4 and 6.95 for MxFP4 under the direct-cast flow.

Watch

Extended reading notes

Core claim

The central claim is that the three identified deficiencies of low-bit MxFP - poor tracking of the largest values in a block, vacant quantization levels, and wasted sign-magnitude codes - have concrete fixes that fit inside the same storage envelope. By placing a 2-bit mantissa on the shared exponent, letting each block select its own format via an index bit, and recycling the -0 code to a useful value, the paper obtains a format family that outperforms state-of-the-art MxFP by up to 0.64 in perplexity and up to 30% in accuracy on MMLU benchmarks while reducing memory footprint by up to 16% at comparable perplexity. The improvements are shown on weight-only and weight-plus-KV-cache quantization of seven modern LLMs, with the largest gains appearing at 4 bits.

Load-bearing premise

The gains from Code Recycling depend on the value half of the smallest quantization step being near-optimal for every model, bitwidth, and block size; the paper tunes this constant on Llama2, Llama3, and Llama3.1, and if the optimum shifts on other model families the reported 4.7% MSE contribution would degrade.

Editorial extensions

If this is right

  • A 5-bit NxFP model can stand in for a 6-bit MxFP model: it matches or beats MxFP6 perplexity on Llama3 and Llama2 while cutting weight-plus-KV-cache footprint by 0.78 to 1.02 GB (13-16%).
  • At 4 bits, NxFP narrows the gap to FP16 substantially: on Llama3, perplexity degradation drops from 0.81 (MxFP4) to 0.43 (NxFP4).
  • The three techniques stack additively: NanoMantissa contributes up to 26% MSE reduction, Adaptive Microexponent another 14%, and Code Recycling up to 4.7%.
  • NxFP keeps the Microscaling deployment model: it works with on-the-fly dequantization on off-the-shelf hardware such as Google TPU and Intel Xeon 6, so no new arithmetic units are required.
  • Because NxFP is a direct-cast format, its perplexity gains are achieved without calibration data, and the format is compatible with calibration-based post-training quantization methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The fixed recycled code value leaves a free hardware/software knob: a future variant could store the recycled value per block or derive it per tensor, potentially recovering extra MSE on model families where the global half-step value is not optimal.
  • The same three mechanisms should apply to activation and KV-cache quantization; the paper demonstrates KV-cache results only on Llama2 and Llama3, but the techniques are formulated for any Microscaling-family vector.
  • The Adaptive Microexponent bit raises a systems question the paper does not fully answer: whether a block-level format bit creates uneven work for SIMD decoders or memory alignment, and whether a prefetch-aware layout recovers those costs.
  • If 4-bit NxFP stabilizes across a wider model zoo, it becomes plausible to deploy 4-bit weights with 16-bit activations on memory-bound inference servers, which would roughly halve weight traffic relative to current 6-bit practices.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper identifies three limitations of low-bit MxFP for direct-cast LLM compression—poor outlier tracking, vacant quantization levels, and wasted sign-magnitude codes—and proposes Nanoscaling Floating-Point (NxFP), which adds a 2-bit NanoMantissa to the shared scaling factor, an adaptive per-block choice between MxFP and BFP microexponents, and code recycling that remaps the -0 pattern to half the smallest quantization step. The authors evaluate NxFP on Llama2, Llama3, Llama3.1, Phi3, Gemma2, and Mistral, reporting quantization MSE, Wikitext2 perplexity, memory-footprint trade-offs, and MMLU accuracy degradation. The headline claims are that NxFP outperforms MxFP by up to 0.64 in perplexity and up to 30% in MMLU accuracy, with NxFP5 matching MxFP6 at up to 16% smaller memory footprint.

Significance. If the results are supported, this is a useful Pareto improvement for low-bit format-based direct-cast compression: the cumulative ablations in Figure 8 and Table 1 are internally consistent in direction across several model families, the dequantization flow in Section 6 is concrete, and NxFP4 substantially narrows the 4-bit gap relative to MxFP4. The main caveats are that the code-recycling parameter is fitted on models that also appear in the evaluation, the MMLU claim rests on a single subject subset, and the reported error-reduction percentages are not internally consistent. These issues are fixable but are load-bearing for the specific quantitative claims.

major comments (3)
  1. [§4.3, §7.6, Table 1, Fig. 9] The code-recycling constant, v_special = half of the smallest quantization step, is selected by perplexity sweeps on Llama3-8B (Fig. 11) and, according to §4.3, on Llama2, Llama3, and Llama3.1. These are the same model families that dominate the main evaluation tables and figures, and no sweep is reported for Mistral, Phi3, Gemma2, for 3/5/6-bit formats, or for block sizes other than 32. The claimed CR contribution (up to 4.7% MSE reduction, §7.2) and the NxFP5-versus-MxFP6 footprint comparison (Figs. 9a and 9b) therefore rely on a value fitted to the evaluation set. Please add transfer sweeps on held-out families and bitwidths, or derive the value from the format's error model; if the optimal value shifts by even one quantization step, the reported gains may not transfer.
  2. [Abstract and §7.5] The abstract claims improvement "by up to 30% in accuracy on MMLU benchmarks," but §7.5 evaluates only the "MMLU-SocialScience" subset and reports accuracy degradation relative to FP16, not accuracy on the full 57-task MMLU benchmark. A single subject subset is not sufficient support for the abstract's MMLU claim. Please report full MMLU accuracy with the few-shot setup and variance, or revise the abstract and conclusion to say "MMLU-SocialScience."
  3. [§7.2 and Figure 8] The error-reduction numbers are internally inconsistent. The text of §7.2 says NxFP4 reduces quantization error by up to 45%, with component reductions of 26%, 14%, and 4.7%, while the caption of Figure 8 says NxFP4 reduces quantization error by 10%–14%. If the 26%, 14%, and 4.7% reductions are relative to the MxFP error and multiplicative, the cumulative reduction is about 39%, not 45%; if they are additive, the caption does not match. Given that the MSE ablation is the primary evidence for the individual techniques, please provide the exact MSE values and a precise definition of the reported percentages.
minor comments (5)
  1. [§4.3] The text says sign-magnitude wastes two codes (-0 and +0), but the recycling scheme remaps only code 10...0; please clarify whether +0 is retained and how the remapped value interacts with the sign bit in the dequantization formula.
  2. [§4.2 and Fig. 5] The adaptive microexponent index bit is described as per vector, but the paper does not state whether this bit is included in the memory-footprint calculations; if it is stored per 32-element block, its overhead should be accounted for or explicitly declared negligible.
  3. [§7.2 and Fig. 8] Figure 8 reports "Normalized Quant Error" and the text gives percentage MSE reductions, but the exact MSE values and the number of blocks or vectors averaged are not reported; please include the underlying numbers and define the normalization.
  4. [§7.1] CommonSenseQA is listed as an evaluation benchmark in §7.1, but no CommonSenseQA results appear in Sections 7 or 8; either add the results or remove the mention.
  5. [§7.1] For the MxFP baselines, the paper says it evaluates different microexponent configurations but does not specify which configuration was used for each bitwidth; stating the exact E/M choice would make the comparison reproducible.

Circularity Check

1 steps flagged · score 3.0 of 10

Code Recycling's remapped value is tuned on the same Llama2/Llama3/Llama3.1 models used for headline results, making part of the reported gain an in-sample fit.

  1. fitted input called prediction [Section 4.3 (Code Recycling) and Section 7.6 (Sweeping the Remapped Value); results in Table 1 and Figures 9-10.]
    "We set the remapped value to be 1/2 × Vsmallest based on our empirical profiling of three representative LLMs (Llama2, Llama3, and Llama3.1)."

    The only free parameter of Code Recycling, the remapped value, is chosen by sweeping perplexity on Llama2, Llama3, and Llama3.1 (Section 7.6, Figure 11). The same three model families appear in the main evaluation tables (Table 1) and in the perplexity-to-footprint plots (Figures 9a-9d), including the headline 0.64 perplexity improvement on Llama3-8B and the reported Code Recycling MSE reduction of 4.7% (Section 7.2). Thus the claimed benefit of this technique is measured on the very data used to set its parameter, so it is an in-sample fit rather than an out-of-sample prediction. The paper does not sweep the value on Mistral, Phi3, Gemma2, or at 3/5/6 bits, so transferability is assumed, not demonstrated. This is a partial fitted-input problem, not a definitional equivalence.

full rationale

The NxFP design is not definitionally circular: NanoMantissa and Adaptive Microexponent are per-vector MSE-based encoding choices (Algorithm 1), not parameters fitted to the evaluation metric, and the full format is also evaluated on Phi3, Mistral, and Gemma2 families that were not used to select the recycled value. However, the Code Recycling constant is an empirical prior selected by perplexity sweeps on Llama2, Llama3, and Llama3.1 (Section 4.3, Section 7.6), and those same families dominate the central quantitative claims: the 0.64 perplexity improvement (Figure 9b), the 4.7% MSE reduction attributed to Code Recycling (Section 7.2), and the 16% footprint reduction at comparable perplexity. Because the recycled-value optimum is not swept on the held-out families or at other bitwidths, the portion of the headline gain attributable to Code Recycling is in-sample for the Llama-family results. This is a genuine selection-on-test-set concern but not a full circularity: the central NxFP-versus-MxFP comparison retains independent out-of-sample content from Phi3, Mistral, and Gemma2, and the other two techniques are not fitted to the evaluation data. I therefore assign a score of 3 instead of a higher partial-circularity score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on three empirical choices: the 1/2 recycled value (fit to perplexity on three evaluated models), the 2-bit NanoMantissa width, and the 32-element block inherited from the MX standard. The core format design is not circular, but the code-recycling parameter is tuned on part of the evaluation set.

free parameters (3)
  • Code-recycling value (v_special / Vsmallest ratio) = 0.5 (half of smallest quantization value)
    Chosen by perplexity sweep on Llama2, Llama3, Llama3.1 in Section 7.6 (Figure 11), then applied to all models and bitwidths.
  • NanoMantissa width = 2 bits
    Set by design, not swept. It is a hand-chosen precision/overhead trade-off that affects the outlier-tracking benefit.
  • Block size = 32 elements
    Uses the OCP MX standard block size; not swept except in Section 7.7 where 8-128 are shown as a trade-off, so the default is an inherited choice.
assumptions (4)
  • domain assumption The three profiled failure modes (outlier tracking, vacant levels, wasted zero codes) are the dominant causes of low-bit MxFP perplexity loss.
    States in Section 3 based on profiling five LLMs; the fixes target exactly these three and do not address other possible causes, such as activation quantization or correlations between weights.
  • domain assumption Direct-cast quantization without calibration is the right measure of format quality for this comparison.
    Section 7.1 explicitly restricts evaluation to direct-cast inference to 'avoid overfitting to calibration set'; this excludes calibrated PTQ methods like GPTQ or AWQ from comparison.
  • domain assumption Perplexity on Wikitext2 and MMLU-SocialScience are sufficient proxies for LLM quality at low bitwidths.
    Evaluations use only these two tasks; no downstream or generative quality metrics are reported.
  • ad hoc to paper The remapped code value is constant across formats (MxFP and BFP) and across bitwidths.
    Section 7.6 sweeps only 4-bit MxFP and BFP on Llama3; the value is then used for all bitwidths and models without further validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Nanoscaling Floating-Point (NxFP): NanoMantissa, Adaptive Microexponents, and Code Recycling for Direct-Cast Compression of Large Language Models." pith.science (2026). https://pith.science/paper/LIXEWA2S

@misc{pith2026241219821,
  author       = {Pith},
  title        = {Pith review of: Nanoscaling Floating-Point (NxFP): NanoMantissa, Adaptive Microexponents, and Code Recycling for Direct-Cast Compression of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LIXEWA2S}},
  note         = {Machine review of arXiv:2412.19821}
}
read the original abstract

As cutting-edge large language models (LLMs) continue to transform various industries, their fast-growing model size and sequence length have led to memory traffic and capacity challenges. Recently, AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm have proposed a Microscaling standard (Mx), which augments block floating-point with microexponents to achieve promising perplexity-to-footprint trade-offs. However, the Microscaling suffers from significant perplexity degradation on modern LLMs with less than six bits. This paper profiles modern LLMs and identifies three main challenges of low-bit Microscaling format, i.e., inaccurate tracking of outliers, vacant quantization levels, and wasted binary code. In response, Nanoscaling (NxFP) proposes three techniques, i.e., NanoMantissa, Adaptive Microexponent, and Code Recycling to enable better accuracy and smaller memory footprint than state-of-the-art MxFP. Experimental results on direct-cast inference across various modern LLMs demonstrate that our proposed methods outperform state-of-the-art MxFP by up to 0.64 in perplexity and by up to 30% in accuracy on MMLU benchmarks. Furthermore, NxFP reduces memory footprint by up to 16% while achieving comparable perplexity as MxFP.

Figures

Figures reproduced from arXiv: 2412.19821 by the authors.

Figure 1
Figure 1. (a) Block FP, (b) Microscaling FP, and (c) our Nanoscal￾ing FP (NxFP). NxFP proposes NanoMantissa, Adaptive MicroEx￾ponent, and Code Recycling to outperform the MxFP standard. sequence length. However, the growth rate of model size (410×/2years) dramatically outweighs the scaling speed of DRAM traffic (1.4×/2years) and capacity (2×/2years), lead￾ing to a ”memory wall” challenge (Gholami et al., 2024). To reduce the … view at source ↗
Figure 2
Figure 2. Visualizing quantization of a real FP16 vector using MxFP. The shared exponent tracks the largest exponent value in each block, and the microexponents track the exponent offset relative the shared exponent. 2. Understanding Microscaling (Mx) Standard Open Compute Project (OCP) has defined four stan￾dards in the Microscaling family, i.e., MxINT8, MxFP8 (E5M2/E4M3), MxFP6 (E3M2/E2M3), and MxFP4 (E2M1) (Rouhani et al.,… view at source ↗
Figure 4
Figure 4. (a) MxFP4 and (b) MxFP4 with NanoMantissa. The proposed NanoMantissa enable MxFP4 with greater precision so that it can track the largest value more accurately [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (7 more)
Figure 6
Figure 6. Figure 6: The flow to apply code recycling on MxFP4. Consider￾ing the perplexity improvement and the implementation overheads, we remaps -0 (code: 1000) to 1 2 Vsmallest. 5. Quantization Algorithm Similar to MxFP standard, NxFP format is not limited to just one quantization algo…
Figure 7
Figure 7. Figure 7: Six steps to decompress Nanoscaling format and run on off-the-shelf hardware. The only difference of NxFP, compared to Mx, is to slice fields, remap wasted code, and multiply NanoMantissa with per-element mantissa. Algorithm 1 MSE-based Quantization Algorithm Input: Fu…
Figure 8
Figure 8. Figure 8: NxFP4 consistently reduces 10%∼14% quantization er￾ror over MxFP4. datasets, parameter counts, number of operations, and model architectures. Baselines. Our primary baseline is Microscaling (Rouhani et al., 2023a) because it can achieve state-of-the-art trade￾offs and …
Figure 9
Figure 9. Figure 9: The perplexity-to-footprint trade-offs of weight-only quantization and weight&KVcache quantization on (a)(b) Llama3 (8B) and (c)(d) Llama2 (7B). The NxFP consistently outperforms MxFP6 in perplexity-to-footprint trade-offs. The sequence length is set to 2K for all expe…
Figure 10
Figure 10. Figure 10: summarizes the accuracy degradation of various LLMs (Llama3, Llama3.1, Gemma2, Llama2, and Mistral) -60 -50 -40 -30 -20 -10 0 BFP6 MxFP6 NxFP6 BFP5 MxFP5 NxFP5 BFP4 MxFP4 NxFP4 BFP3 MxFP3 NxFP3 Accuracy Degradation (%) on MMLU-Social Gemma2-2B Llama2-7B Llama3-8B Llam…
Figure 11
Figure 11. Figure 11: Perplexity of different recycled values to remap wasted code 10...0 on (a) MxFP4 and (b) BFP4 on Llama3 (8B). 6.0 6.4 6.8 7.2 7.6 8.0 3.5 4 4.5 5 Perplexity Weight Footprint (GB) BFP4 MxFP4 NxFP4 (NM) NxFP4 (NM+AM) NxFP4 (NM+AM+CR) BS=128 BS=64 BS=32 BS=16 BS=8 Better…
Figure 12
Figure 12. Figure 12: Perplexity-to-footprint trade-offs across different block sizes using 4 bits. 7.6. Sweeping the Remapped Value of Code Recycling We sweep the perplexity of different remapped value on MxFP and BFP. Although sweeping through all remapped value is possible, we consider …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference

    cs.AR 2026-08 conditional novelty 6.0 of 10

    AdaMX puts per-block format and enhancement metadata into spare MX exponent bits, recovering 83% of MXFP4's commonsense accuracy loss at the same 4-bit storage budget.

Reference graph

Works this paper leans on

35 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    A., Awan, A

    Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Mendes, C. C. T., Chen, W., Chaudhary, V., Chopra, P., Giorno, A. D., de Rosa, G., Dixon, M., Eldan, R., Iter, D., Garg, A., Goswami, A., Gunasekar, S., Haider, E., Hao, J....

    work page 2024

  3. [3]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert - Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford...

    arXiv 2020

  4. [4]

    Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point

    Darvish Rouhani, B., Lo, D., Zhao, R., Liu, M., Fowers, J., Ovtcharov, K., Vinogradsky, A., Massengill, S., Yang, L., Bittner, R., Forin, A., Zhu, H., Na, T., Patel, P., Che, S., Chand Koppaka, L., Song, X., Som, S., Das, K., T, S., Reinhardt, S., Lanka, S., Chung, E., and Burger, D. Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with M...

    work page 2020

  5. [5]

    S., Naghshineh, S., Park, J., and Naumov, M

    Darvish Rouhani, B., Zhao, R., Elango, V., Shafipour, R., Hall, M., Mesmakhosroshahi, M., More, A., Melnick, L., Golub, M., Varatkar, G., Shao, L., Kolhe, G., Melts, D., Klar, J., L'Heureux, R., Perry, M., Burger, D., Chung, E., Deng, Z. S., Naghshineh, S., Park, J., and Naumov, M. With Shared Microexponents, A Little Shifting Goes a Long Way . In Proceed...

    arXiv 2023

  6. [6]

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra...

    arXiv 2024

  7. [7]

    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers . In International Conference on Learning Representation (ICLR), 2023

    work page 2023

  8. [8]

    A framework for few-shot language model evaluation, 07 2024

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL http...

    arXiv 2024

Show all 35 references
  1. [9]

    W., and Keutzer, K

    Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W., and Keutzer, K. AI and Memory Wall . IEEE Micro, 2024. doi:10.1109/MM.2024.3373763

    2024

  2. [10]

    A., Welbl, J., Clark, A., et al

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. An Empirical Analysis of Compute-Optimal Large Language Model Training . Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  3. [11]

    W., Shao, Y

    Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length llm inference with kv cache quantization, 2024

    2024

  4. [12]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7B . In ArXiv, 2023

    2023

  5. [13]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. CoRR, 2020

    2020

  6. [14]

    J., Wang, X., Nassar, M., Bansal, A

    K\" o ster, U., Webb, T. J., Wang, X., Nassar, M., Bansal, A. K., Constable, W. H., Elibol, O. H., Gray, S., Hall, S., Hornof, L., Khosrowshahi, A., Kloss, C., Pai, R. J., and Rao, N. Flexpoint: an adaptive numerical format for efficient training of deep neural networks. In Pr...

    2017

  7. [15]

    B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., Liu, Q., Zheltonozhskii, E., Zhuo, T

    Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., Liu, Q., Zheltonozhskii, E., Zhuo, T. Y., Wang, T., Dehaene, O., Davaadorj, M., Lamy-Poirier, J., Monteiro, J., Shliazhko, O., Gontier, N., Meade, N., Zebaze, A., Yee...

    2023

  8. [16]

    AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

    Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration . In MLSys, 2024

    2024

  9. [17]

    and Liu, R.-S

    Lo, Y.-C. and Liu, R.-S. Bucket getter: A bucket-based processing engine for low-bit block floating point (bfp) dnns. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO '23, pp.\ 1002–1015, New York, NY, USA, 2023. Association for Co...

    2023

  10. [18]

    Block and subword-scaling floating-point ( BSFP ) : An efficient non-uniform quantization for low precision inference

    Lo, Y.-C., Lee, T.-K., and Liu, R.-S. Block and subword-scaling floating-point ( BSFP ) : An efficient non-uniform quantization for low precision inference. In The Eleventh International Conference on Learning Representations (ICLR), 2023 a . URL https://openreview.net/forum?i...

    2023

  11. [19]

    Lv: Latency-versatile floating-point engine for high-performance deep neural networks

    Lo, Y.-C., Tsai, Y.-C., and Liu, R.-S. Lv: Latency-versatile floating-point engine for high-performance deep neural networks. IEEE Computer Architecture Letters, 22 0 (2): 0 125--128, 2023 b . doi:10.1109/LCA.2023.3287096

    2023

  12. [20]

    P., Mallick, R., Wollaber, A

    Macdonald, J. P., Mallick, R., Wollaber, A. B., Pe\ n a, J. D., McNeese, N., and Siu, H. C. Language, Camera, Autonomy! Prompt-engineered Robot Control for Rapidly Evolving Deployment . In ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2024. ISBN 979840070...

    2024

  13. [21]

    Intel Unleashes Enterprise AI with Gaudi 3, AI Open Systems Strategy and New Customer Wins

    Mann, D. Intel Unleashes Enterprise AI with Gaudi 3, AI Open Systems Strategy and New Customer Wins . In Intel, 2024. Accessed: 2024-05-05

    2024

  14. [22]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models . ArXiv, 2016

    2016

  15. [23]

    Introducing Meta Llama 3: The most capable openly available LLM to date , 2024

    Meta . Introducing Meta Llama 3: The most capable openly available LLM to date , 2024. URL https://ai.meta.com/blog/meta-llama-3/

    2024

  16. [24]

    NVIDIA Blackwell Architecture Technical Brief

    NVIDIA. NVIDIA Blackwell Architecture Technical Brief . 2024. Accessed: 2024-05-06

    2024

  17. [25]

    G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C

    Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S....

    2024 arXiv

  18. [26]

    D., Garegrat, N., Savell, T., More, A., Han, K.-N., Zhao, R., Mathew Hall, J

    Rouhani, B. D., Garegrat, N., Savell, T., More, A., Han, K.-N., Zhao, R., Mathew Hall, J. K., Chung, E., Yu, Y., Schulte, M., Wittig, R., Bratt, I., Stephens, N., Milanovic, J., Brothers, J., Dubey, P., Cornea, M., Heinecke, A., Rodriguez, A., Langhammer, M., Deng, S., Naumov,...

    2023

  19. [27]

    Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., Dusan, S., Elango, V., Golub, M., Heinecke, A., James-Roxby, P., Jani, D., Kolhe, G., Langhammer, M., Li, A., Melnick, L., Mesmakhosroshahi, M., Rodri...

    2023

  20. [28]

    Algorithm-Hardware Co-Design of Adaptive Floating-Point Encodings for Resilient Deep Learning Inference

    Tambe, T., Yang, E.-Y., Wan, Z., Deng, Y., Janapa Reddi, V., Rush, A., Brooks, D., and Wei, G.-Y. Algorithm-Hardware Co-Design of Adaptive Floating-Point Encodings for Resilient Deep Learning Inference . In ACM/IEEE Design Automation Conference (DAC), 2020. doi:10.1109/DAC1807...

    2020

  21. [29]

    LLaMA: Open and Efficient Foundation Language Models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. LLaMA: Open and Efficient Foundation Language Models . In ArXiv, 2023 a

    2023

  22. [30]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N....

    2023

  23. [31]

    How to Quadruple LLM Decoding Performance with Speculative Decoding (SpD) and Microscaling (MX) Formats on Qualcomm Cloud AI 100

    Vaidhyanathan, N. How to Quadruple LLM Decoding Performance with Speculative Decoding (SpD) and Microscaling (MX) Formats on Qualcomm Cloud AI 100 . In Qualcomm's Developer Blog, 2024. Accessed: 2024-05-05

    2024

  24. [32]

    GPTVQ: The Blessing of Dimensionality for LLM Quantization

    van Baalen, M., Kuzmin, A., Nagel, M., Couperus, P., Bastoul, C., Mahurin, E., Blankevoort, T., and Whatmough, P. GPTVQ: The Blessing of Dimensionality for LLM Quantization . In arXiv, 2024

    2024

  25. [33]

    Sg-float: Achieving memory access and computing power reduction using self-gating float in cnns

    Wu, J.-S., Hsu, T.-W., and Liu, R.-S. Sg-float: Achieving memory access and computing power reduction using self-gating float in cnns. 22 0 (6), November 2023. ISSN 1539-9087. doi:10.1145/3624582. URL https://doi.org/10.1145/3624582

    2023 doi

  26. [34]

    Xia, H., Zheng, Z., Wu, X., Chen, S., Yao, Z., Youn, S., Bakhtiari, A., Wyatt, M., Zhuang, D., Zhou, Z., Ruwase, O., He, Y., and Song, S. L. Fp6-llm: Efficiently serving large language models through fp6-centric algorithm-system co-design, 2024

    2024

  27. [35]

    BiE: Bi-Exponent Block Floating-Point for Large Language Models Quantization

    Zou, L., Zhao, W., Yin, S., Bai, C., Sun, Q., and Yu, B. BiE: Bi-Exponent Block Floating-Point for Large Language Models Quantization . In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=DbyHDYslM7

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.