REVIEW 26 cited by
Microscaling Data Formats for Deep Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Narrow bit-width data formats are key to reducing the computational and storage costs of modern deep learning applications. This paper evaluates Microscaling (MX) data formats that combine a per-block scaling factor with narrow floating-point and integer types for individual elements. MX formats balance the competing needs of hardware efficiency, model accuracy, and user friction. Empirical results on over two dozen benchmarks demonstrate practicality of MX data formats as a drop-in replacement for baseline FP32 for AI inference and training with low user friction. We also show the first instance of training generative language models at sub-8-bit weights, activations, and gradients with minimal accuracy loss and no modifications to the training recipe.
Forward citations
Cited by 26 Pith papers
-
When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inference
A simulator predicts that four-way prefill/decode/attention/FFN disaggregation beats unified serving on prefill-heavy agentic workloads by up to 2.06x, but only with stage-specialized custom NPUs and rich enough hardw...
-
One QK Channel, Many Sources: Guarding Low-Precision Attention Collapse
Different low-precision errors converge on the same query-key spectral runaway, entry is gated by temporal sign-coherence, and a dormant query-key normalization guard contains it.
-
Jack of All Scales: A Versatile FPGA Tensor Block for MXFP Precisions
Targeted DSP tensor-mode changes enable native MXFP4/MXFP6/E4M3 support on Agilex-5-like FPGAs, with a 36% block-area cost and 4.2x average systolic-array throughput gain over baseline mapping strategies.
-
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
Full-stack NVFP4 LLM pretraining is stable with a 1.47% loss gap to BF16 via LoRA-SVD projections, transformed AdamW/Root optimizers, and mixed-precision attention.
-
Characterization and Mitigation of Training Instabilities in Microscaling Formats
Block-scaled MX low-precision training is unstable because quantization of tightly clustered layer-norm weights and some activations injects multiplicative gradient bias, and this can be fixed by keeping activations i...
-
FP4 All the Way: Fully Quantized Training of LLMs
A 7-billion parameter LLM trained almost entirely in 4-bit float precision matches a standard 16-bit baseline after a brief quantization-aware fine-tuning phase.
-
MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attention
A closed-form MXFP4 scale boundary Qmax=7.25 plus pre-normalization softmax quantization recovers nearly all FP16 video quality lost by naive MXFP4 attention.
-
dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats
dMX learns continuous per-layer MXFP bit-width offsets with temperature annealing and target-aware regularization, yielding Pareto-better mixed-precision LLMs than uniform or KL-heuristic baselines.
-
Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference
AdaMX puts per-block format and enhancement metadata into spare MX exponent bits, recovering 83% of MXFP4's commonsense accuracy loss at the same 4-bit storage budget.
-
Studying quantization trade-offs for efficient inference deployment in machine translation
Quantized Hy-MT2 models stay accurate at long context, but quantized EuroLLM 9B/22B models collapse (up to ~60% chrF++ drop) while W4A8/W8A8 plus 200–400-token chunking improves serving throughput.
-
LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference
LightRot uses grouped local rotation plus outlier alignment to make 4-bit LLaMA inference accurate and cheap, claiming 27.4 TOPS/W on a 28nm accelerator.
-
GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference
Rotation and fine-grained group quantization can work together if rotation spans several quantization groups and outlier channels are permuted onto harmonic Hadamard rows, enabling 4-bit LLM inference with integer-onl...
-
Stable FP4 Training via Transposition-Invariant Block Quantization
Transposition-invariant 2D block FP4 quantization plus truncation-free scaling and stochastic rounding enables stable end-to-end FP4 LLM training within ~1% of BF16.
-
MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
MXSens allocates 8-bit precision to the 32 most sensitive columns per layer, 6-bit to moderately sensitive columns, and 4-bit elsewhere in MXINT, improving WikiText-2 perplexity over prior 4-bit LLM quantization methods.
-
CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits
CANN Bench is an Ascend-NPU benchmark with 53 operators and 1,060 released test cases that scores agent-generated kernels on compilation, correctness, and hardware-anchored performance.
-
DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce
DynamiQ compresses gradients for multi-hop all-reduce with per-block bit-width allocation, block reordering, and fused recompression, matching BF16 accuracy at ~5 bits per coordinate and improving time-to-accuracy by ...
-
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
By appending quantized residual channels to NVFP4 activations and duplicating the matching weights, ARCQuant reaches W4A8-level accuracy while keeping a single unified 4-bit GEMM.
-
SeVeDo: A Heterogeneous Transformer Accelerator for Low-Bit Inference via Hierarchical Group Quantization and SVD-Guided Mixed Precision
SeVeDo achieves 12.7–13.8 TOPS/W in 28nm for ViT-Base and Llama2-7B by combining SVD-based low-rank decomposition, hierarchical group quantization, and SVD-guided mixed precision.
-
Hybrid Systolic Array Accelerator with Optimized Dataflow for Edge Large Language Model Inference
A 28nm hybrid systolic array accelerator with MXINT4 quantization and fused RMSNorm/RoPE units reports 247/117 token/s/mm2 running RetNet 1.3B, claiming over 2.45x/13.5x area-efficiency gains over prior edge LLM accelerators.
-
Recipes for Pre-training LLMs with MXFP8
MXFP8-E4M3 with round-up scale computation matches BF16 pre-training accuracy for LLMs up to 8B parameters, enabling 2x faster training on Blackwell.
-
FPQVAR: Floating Point Quantization for Visual Autoregressive Model with FPGA Hardware Co-design
A post-training floating-point quantization framework with grouped Hadamard rotation and learned smoothing brings 4-bit visual autoregressive image generation to near-FP16 quality, plus a matching FPGA accelerator.
-
WINT: A Novel Weighted Integer Representation with Improved Error Characteristics
WINT (M imes2^E) with 2–3 exponent bits gives both larger range and lower mean relative error than same-width integers for bit widths of 12 and above.
-
FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling
FOCUS relaxes the tie between quantization and dequantization scales in FP4 formats, learning a full-precision coefficient and sub-block scales to find better FP4 weight assignments.
-
TorchAO: PyTorch-Native Training-to-Serving Model Optimization
TorchAO delivers a PyTorch-native, end-to-end workflow for FP8 training, QAT, PTQ, and sparsity, with benchmarked speedups and production use in quantized Llama releases.
-
Beyond the Buzz: A Pragmatic Take on Inference Disaggregation
Disaggregated serving (separate prefill and decode GPU pools) expands the throughput-interactivity Pareto frontier mainly for prefill-heavy workloads and models larger than about 10B parameters, provided the prefill-t...
-
How to keep pushing ML accelerator performance? Know your rooflines!
A JSSC survey organizing ML accelerator techniques around two roofline curves, one for throughput and one for energy efficiency, with the insight that their bottleneck knees often sit at different arithmetic intensities.
Discussion (0). Sign in to comment.