{"id":"687ddf60-27c4-483c-af56-b6fd30b80380","arxiv_id":"2412.19821","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A Nanoscaling format adds a shared 2-bit mantissa, per-block format choice, and recycled zero codes to beat MxFP on LLM perplexity and memory footprint.","lead":"This paper describes a new low-bit number format for storing large language models in less memory, and tests it on seven models. It matters because cutting memory use by about one sixth at equal quality could make large models cheaper to run on smaller hardware.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The code-recycling constant is tuned on evaluation models; the 4.7% MSE / 0.64-ppl gains hinge on it transferring to held-out families and bitwidths, which §4.3 and §7.6 do not establish.","rationale":"The reader's weakest_assumption is exactly the code-recycling constant's generality, and my reading of §4.3, §7.2, §7.6, Table 1, and Figs. 9-11 confirms that this is the most load-bearing weak point. The strongest_claim's most concrete, novel quantitative selling points — the 4.7% MSE reduction and the NxFP5-at-16%-less-memory-than-MxFP6 result — both include Code Recycling, and the constant is fixed a priori from a sweep on evaluation models. The paper does provide independent support in the form of a quantitative ablation (Fig. 8) and a consistent per-technique breakdown, which I credit; the concern is not that the effect is fabricated but that its size is conditioned on a tuned hyperparameter that has not been shown to transfer. I agree with the reader's CONDITIONAL verdict, because the paper's claims are plausible and the missing artifact/transfer evidence is exactly what the condition should address. My concrete test — sweeping the recycled value on held-out models and bitwidths — would settle whether the CR gains hold; if they do, the verdict should become ACCEPT for the main claims, and if not, the central trade-off claims would need re-scoping. This is the same check the reader implicitly requested (hold-out models, 3/5/6 bits), so I mark 'agree'.","tokens_in":18903,"tokens_out":1250,"duration_ms":11451,"concrete_test":"Re-run the §7.6 sweep of recycled values for code 10...0 on held-out models (Mistral-7B, Phi3-4B, Gemma2-2B) and at NxFP5 and NxFP3 on Llama3-8B, using the same perplexity protocol as Table 1. Record whether the optimal remapped value is within {0, 1/2 × V_smallest, 1.0 × V_smallest, midpoint-to-second-largest}: if the optimum is not 1/2 × V_smallest for any held-out setting, recompute NxFP4/5 gains with the per-setting optimum to bound the CR contribution; the central claim should be reported with that bound.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that NxFP outperforms MxFP by up to 0.64 perplexity and 30% MMLU accuracy, with NxFP5 matching MxFP6 at 16% lower memory. The quantitative story rests on three techniques, of which Code Recycling is the least robust because its only free parameter, the recycled value (1/2 × V_smallest), is selected empirically by sweeping perplexity in §4.3/§7.6. That sweep is reported only for Llama3-8B at 4 bits (Fig. 11), while §4.3 says the choice was based on 'three representative LLMs (Llama2, Llama3, Llama3.1)' — the same three families dominate the main tables (Table 1, Figs. 9-10). The paper never sweeps the remapped value for Mistral, Phi3, Gemma2, or for 3/5/6-bit formats. If the optimum shifts even one quantization step in those settings, the claimed CR contribution (up to 4.7% MSE reduction; §7.2) could shrink or invert, which would also move the headline NxFP5-vs-MxFP6 memory trade (Figs. 9a/9b) because that comparison explicitly includes the CR-enabled point. This is not an accusation of cherry-picking; it is a genuine generalization gap: the parameter is not derived from the format's error model but from a model-specific sweep. The paper itself notes the value is 'empirical' (§4.3), so a transfer test is the missing control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies three limitations of low-bit MxFP for direct-cast LLM compression—poor outlier tracking, vacant quantization levels, and wasted sign-magnitude codes—and proposes Nanoscaling Floating-Point (NxFP), which adds a 2-bit NanoMantissa to the shared scaling factor, an adaptive per-block choice between MxFP and BFP microexponents, and code recycling that remaps the -0 pattern to half the smallest quantization step. The authors evaluate NxFP on Llama2, Llama3, Llama3.1, Phi3, Gemma2, and Mistral, reporting quantization MSE, Wikitext2 perplexity, memory-footprint trade-offs, and MMLU accuracy degradation. The headline claims are that NxFP outperforms MxFP by up to 0.64 in perplexity and up to 30% in MMLU accuracy, with NxFP5 matching MxFP6 at up to 16% smaller memory footprint.","tokens_in":19289,"tokens_out":8237,"duration_ms":70832,"significance":"If the results are supported, this is a useful Pareto improvement for low-bit format-based direct-cast compression: the cumulative ablations in Figure 8 and Table 1 are internally consistent in direction across several model families, the dequantization flow in Section 6 is concrete, and NxFP4 substantially narrows the 4-bit gap relative to MxFP4. The main caveats are that the code-recycling parameter is fitted on models that also appear in the evaluation, the MMLU claim rests on a single subject subset, and the reported error-reduction percentages are not internally consistent. These issues are fixable but are load-bearing for the specific quantitative claims.","major_comments":[{"comment":"The code-recycling constant, v_special = half of the smallest quantization step, is selected by perplexity sweeps on Llama3-8B (Fig. 11) and, according to §4.3, on Llama2, Llama3, and Llama3.1. These are the same model families that dominate the main evaluation tables and figures, and no sweep is reported for Mistral, Phi3, Gemma2, for 3/5/6-bit formats, or for block sizes other than 32. The claimed CR contribution (up to 4.7% MSE reduction, §7.2) and the NxFP5-versus-MxFP6 footprint comparison (Figs. 9a and 9b) therefore rely on a value fitted to the evaluation set. Please add transfer sweeps on held-out families and bitwidths, or derive the value from the format's error model; if the optimal value shifts by even one quantization step, the reported gains may not transfer.","section":"§4.3, §7.6, Table 1, Fig. 9"},{"comment":"The abstract claims improvement \"by up to 30% in accuracy on MMLU benchmarks,\" but §7.5 evaluates only the \"MMLU-SocialScience\" subset and reports accuracy degradation relative to FP16, not accuracy on the full 57-task MMLU benchmark. A single subject subset is not sufficient support for the abstract's MMLU claim. Please report full MMLU accuracy with the few-shot setup and variance, or revise the abstract and conclusion to say \"MMLU-SocialScience.\"","section":"Abstract and §7.5"},{"comment":"The error-reduction numbers are internally inconsistent. The text of §7.2 says NxFP4 reduces quantization error by up to 45%, with component reductions of 26%, 14%, and 4.7%, while the caption of Figure 8 says NxFP4 reduces quantization error by 10%–14%. If the 26%, 14%, and 4.7% reductions are relative to the MxFP error and multiplicative, the cumulative reduction is about 39%, not 45%; if they are additive, the caption does not match. Given that the MSE ablation is the primary evidence for the individual techniques, please provide the exact MSE values and a precise definition of the reported percentages.","section":"§7.2 and Figure 8"}],"minor_comments":[{"comment":"The text says sign-magnitude wastes two codes (-0 and +0), but the recycling scheme remaps only code 10...0; please clarify whether +0 is retained and how the remapped value interacts with the sign bit in the dequantization formula.","section":"§4.3"},{"comment":"The adaptive microexponent index bit is described as per vector, but the paper does not state whether this bit is included in the memory-footprint calculations; if it is stored per 32-element block, its overhead should be accounted for or explicitly declared negligible.","section":"§4.2 and Fig. 5"},{"comment":"Figure 8 reports \"Normalized Quant Error\" and the text gives percentage MSE reductions, but the exact MSE values and the number of blocks or vectors averaged are not reported; please include the underlying numbers and define the normalization.","section":"§7.2 and Fig. 8"},{"comment":"CommonSenseQA is listed as an evaluation benchmark in §7.1, but no CommonSenseQA results appear in Sections 7 or 8; either add the results or remove the mention.","section":"§7.1"},{"comment":"For the MxFP baselines, the paper says it evaluates different microexponent configurations but does not specify which configuration was used for each bitwidth; stating the exact E/M choice would make the comparison reproducible.","section":"§7.1"}],"recommendation":"major_revision","confidential_remarks":"The core format idea is plausible and the ablation structure is a strength, but the code-recycling parameter is fitted on models that then appear in the headline tables, and the MMLU claim is based on one subject subset. Both are load-bearing for the paper's central quantitative claims and should be addressed before acceptance. I would not reject on these grounds, as both are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious look. What's new is the specific combination of three simple ideas into one format: a shared 2-bit NanoMantissa, a per-block format-selection bit between MxFP and BFP, and recycling the wasted -0 code to half the smallest step. Individually each idea is small, but the cumulative NxFP format is not in the OCP Mx spec, and the ablation study in Figure 8 and Table 1 gives a clear, monotonic story. The headline claim that NxFP5 matches MxFP6 perplexity with 13-16% less memory is plausible and useful for direct-cast inference.\n\nThe paper does several things well. The evaluation spans six models from different families, and the cross-model trend is stable. The quantization algorithm is given in reasonable detail, and the on-the-fly dequantization flow is concrete. The authors are also honest that the code-recycling constant is chosen empirically rather than derived.\n\nThe soft spots are real but not disqualifying. The recycled-value parameter is swept on Llama2/Llama3/Llama3.1, and those same models dominate the main tables. The paper never shows that 1/2 × V_smallest transfers to Mistral, Gemma2, Phi3, or to 3-, 5-, and 6-bit formats. The stress-test note is correct that this is a generalization gap, and because the CR contribution is up to 4.7% MSE, the headline memory trade partially rests on it. That said, the paper explicitly labels the value as empirical, so the missing control is a transfer experiment, not a conceptual flaw. Two smaller issues: the MMLU result is only on the Social Sciences subset, which is a narrow basis for the 30% accuracy claim, and there is no comparison with BSFP or BiE, which are cited but not evaluated. No error bars or code release, so artifact availability matters.\n\nThe central argument holds together. The format design is coherent, the reported gains are consistent, and the limitations are the kind a revision can address.\n\nMy recommendation: send it to peer review. Ask for code or artifacts, the full MMLU or at least several subsets, and a sweep of the recycled value on held-out models and bitwidths. If the constant transfers, this is a useful systems result for 4-bit LLM inference.","headline":"Solid, incremental contribution to low-bit LLM quantization: the NxFP format combination is new, the ablations are honest, and the main soft spot is a tuned constant that needs a transfer test.","tokens_in":19806,"tokens_out":1289,"would_cite":true,"duration_ms":13008,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NxFP, a new Nanoscaling format, beats MxFP on LLM perplexity while using up to 16 percent less memory.","keywords":["Nanoscaling Floating-Point","Microscaling (Mx)","low-bit quantization","direct-cast inference","large language models","perplexity","MMLU","code recycling"],"falsifier":"Quantize Gemma2-2B and Mistral-7B at 4 bits with NxFP while sweeping the recycled code value over the grid {0, 0.25, 0.5, 0.75, 1} times the smallest quantization step, and compare Wikitext-2 perplexity with the fixed half-step choice; if a value other than 0.5 yields meaningfully lower perplexity on these models, the single-constant design is not generally optimal. A second check is to reproduce the reported 4-bit Llama3-8B perplexities of 6.57 for NxFP4 and 6.95 for MxFP4 under the direct-cast flow.","tokens_in":18706,"feed_emoji":"📉","tokens_out":6447,"duration_ms":50410,"temperature":0.7,"pith_summary":"This paper proposes Nanoscaling Floating-Point (NxFP), a number format for compressing large language models by direct-cast quantization. NxFP augments the Microscaling (MxFP) standard with three techniques: a shared 2-bit NanoMantissa that tracks outlier magnitudes per block, an Adaptive Microexponent bit that lets each block choose between MxFP and block-floating-point layout, and Code Recycling that maps the wasted -0 code to half the smallest quantization step. The paper claims NxFP beats MxFP by up to 0.64 in Wikitext-2 perplexity and up to 30% in MMLU accuracy, and that a 5-bit NxFP model matches 6-bit MxFP perplexity with 13-16% less memory footprint. If correct, this gives sub-6-bit Microscaling formats a path to be usable for generative inference.","feed_headline":"NxFP beats MxFP on LLM perplexity, cutting memory 16%","feed_subtitle":"A shared 2-bit mantissa, adaptive microexponents, and recycled codes make sub-6-bit quantization viable.","key_machinery":"The central object is the NxFP vector format, five fields per block of 32 elements: sign, element mantissa, shared exponent, microexponent (or its absence), and a format bit. The NanoMantissa is a shared 2-bit mantissa stored alongside the shared exponent that scales the whole block by a value $1.m$, letting the format represent values above the largest MxFP code; the Adaptive Microexponent is a single index bit that decides, per block, between the MxFP layout (good for scattered distributions) and the BFP layout (good for clustered distributions); Code Recycling remaps the sign-magnitude code $10...0$, which would be $-0$, to half of the smallest quantization step, recoverable in hardware by a one-bit right shift. The quantization algorithm exercises each choice - NanoMantissa on or off and MxFP or BFP - and picks the combination minimizing MSE, so the three techniques act as a small per-block search over formats. These mechanisms add no bits to the element stream; they only redistribute existing storage.","core_discovery":"The central claim is that the three identified deficiencies of low-bit MxFP - poor tracking of the largest values in a block, vacant quantization levels, and wasted sign-magnitude codes - have concrete fixes that fit inside the same storage envelope. By placing a 2-bit mantissa on the shared exponent, letting each block select its own format via an index bit, and recycling the -0 code to a useful value, the paper obtains a format family that outperforms state-of-the-art MxFP by up to 0.64 in perplexity and up to 30% in accuracy on MMLU benchmarks while reducing memory footprint by up to 16% at comparable perplexity. The improvements are shown on weight-only and weight-plus-KV-cache quantization of seven modern LLMs, with the largest gains appearing at 4 bits.","pith_inferences":["The fixed recycled code value leaves a free hardware/software knob: a future variant could store the recycled value per block or derive it per tensor, potentially recovering extra MSE on model families where the global half-step value is not optimal.","The same three mechanisms should apply to activation and KV-cache quantization; the paper demonstrates KV-cache results only on Llama2 and Llama3, but the techniques are formulated for any Microscaling-family vector.","The Adaptive Microexponent bit raises a systems question the paper does not fully answer: whether a block-level format bit creates uneven work for SIMD decoders or memory alignment, and whether a prefetch-aware layout recovers those costs.","If 4-bit NxFP stabilizes across a wider model zoo, it becomes plausible to deploy 4-bit weights with 16-bit activations on memory-bound inference servers, which would roughly halve weight traffic relative to current 6-bit practices."],"forward_implications":["A 5-bit NxFP model can stand in for a 6-bit MxFP model: it matches or beats MxFP6 perplexity on Llama3 and Llama2 while cutting weight-plus-KV-cache footprint by 0.78 to 1.02 GB (13-16%).","At 4 bits, NxFP narrows the gap to FP16 substantially: on Llama3, perplexity degradation drops from 0.81 (MxFP4) to 0.43 (NxFP4).","The three techniques stack additively: NanoMantissa contributes up to 26% MSE reduction, Adaptive Microexponent another 14%, and Code Recycling up to 4.7%.","NxFP keeps the Microscaling deployment model: it works with on-the-fly dequantization on off-the-shelf hardware such as Google TPU and Intel Xeon 6, so no new arithmetic units are required.","Because NxFP is a direct-cast format, its perplexity gains are achieved without calibration data, and the format is compatible with calibration-based post-training quantization methods."],"supporting_citations":[{"why":"Defines the OCP Microscaling formats, including the per-block shared exponent plus microexponent structure and the block size of 32, which NxFP extends and is compared against.","marker":"(Rouhani et al., 2023a)"},{"why":"Reports the MxFP perplexity numbers on Llama models at low bitwidths that motivate the work and serve as the primary comparison baseline.","marker":"(Rouhani et al., 2023b)"},{"why":"Sharpens the shared-microexponent construction that NxFP builds on by adding NanoMantissa and code recycling.","marker":"(Darvish Rouhani et al., 2023)"},{"why":"Provides the block floating-point (BFP) format that NxFP uses as the alternative chosen by the Adaptive Microexponent index bit.","marker":"(Darvish Rouhani et al., 2020)"},{"why":"Supplies the Wikitext-2 dataset used for all perplexity evaluations in the paper.","marker":"(Merity et al., 2016)"},{"why":"Supplies the lm-eval-harness framework used to compute MMLU and CommonsenseQA accuracy.","marker":"(Gao et al., 2024)"},{"why":"Provides the KVquant codebase that the paper extends to evaluate weight-plus-KV-cache quantization.","marker":"(Hooper et al., 2024)"},{"why":"Provides the GPTVQ codebase extended for the direct-cast quantization flow and comparisons.","marker":"(van Baalen et al., 2024)"}],"fun_headline_variants":["NxFP: 0.64 perplexity gain, 30% MMLU boost, 16% memory cut","Nanoscaling FP outdoes MxFP: better LLM accuracy, smaller memory","NxFP: sub-6-bit quantization with 30% accuracy lift and 16% memory off","Recycling code and mantissas: NxFP beats MxFP on LLM compression","NxFP shows how to fix low-bit MxFP: 30% accuracy, 16% memory win"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gains from Code Recycling depend on the value half of the smallest quantization step being near-optimal for every model, bitwidth, and block size; the paper tunes this constant on Llama2, Llama3, and Llama3.1, and if the optimum shifts on other model families the reported 4.7% MSE contribution would degrade.","fun_headline_variants_meta":{"raw":{"variants":["NxFP: 0.64 perplexity gain, 30% MMLU boost, 16% memory cut","Nanoscaling FP outdoes MxFP: better LLM accuracy, smaller memory","NxFP: sub-6-bit quantization with 30% accuracy lift and 16% memory off","Recycling code and mantissas: NxFP beats MxFP on LLM compression","NxFP shows how to fix low-bit MxFP: 30% accuracy, 16% memory win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00061,"raw_usage":{"total_tokens":2849,"prompt_tokens":967,"completion_tokens":1882,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":1751}},"tokens_in":583,"tokens_out":1882,"duration_ms":10290,"temperature":1.0,"reasoning_tokens":1751,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:03:30.675313+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Quantize Gemma2-2B and Mistral-7B at 4 bits with NxFP while sweeping the recycled code value over the grid {0, 0.25, 0.5, 0.75, 1} times the smallest quantization step, and compare Wikitext-2 perplexity with the fixed half-step choice; if a value other than 0.5 yields meaningfully lower perplexity on these models, the single-constant design is not generally optimal. A second check is to reproduce the reported 4-bit Llama3-8B perplexities of 6.57 for NxFP4 and 6.95 for MxFP4 under the direct-cast flow.","supporting_citations":[{"cited_title":"Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point","cited_arxiv_id":null,"evidence_quote":"Provides the block floating-point (BFP) format that NxFP uses as the alternative chosen by the Adaptive Microexponent index bit."},{"cited_title":"Pointer sentinel mixture models","cited_arxiv_id":null,"evidence_quote":"Supplies the Wikitext-2 dataset used for all perplexity evaluations in the paper."},{"cited_title":"W., Shao, Y","cited_arxiv_id":null,"evidence_quote":"Provides the KVquant codebase that the paper extends to evaluate weight-plus-KV-cache quantization."},{"cited_title":"GPTVQ: The Blessing of Dimensionality for LLM Quantization","cited_arxiv_id":null,"evidence_quote":"Provides the GPTVQ codebase extended for the direct-cast quantization flow and comparisons."}],"review_version":1}