REVIEW 7 cited by
Training LLMs with MXFP4
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Low precision (LP) datatypes such as MXFP4 can accelerate matrix multiplications (GEMMs) and reduce training costs. However, directly using MXFP4 instead of BF16 during training significantly degrades model quality. In this work, we present the first near-lossless training recipe that uses MXFP4 GEMMs, which are $2\times$ faster than FP8 on supported hardware. Our key insight is to compute unbiased gradient estimates with stochastic rounding (SR), resulting in more accurate model updates. However, directly applying SR to MXFP4 can result in high variance from block-level outliers, harming convergence. To overcome this, we use the random Hadamard tranform to theoretically bound the variance of SR. We train GPT models up to 6.7B parameters and find that our method induces minimal degradation over mixed-precision BF16 training. Our recipe computes $>1/2$ the training FLOPs in MXFP4, enabling an estimated speedup of $>1.3\times$ over FP8 and $>1.7\times$ over BF16 during backpropagation.
Forward citations
Cited by 7 Pith papers
-
HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models
In end-to-end 4-bit RL post-training, rollout activation underflow, not training quantization, is the main accuracy killer; a sparse residual correction closes most of the gap to BF16.
-
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
Full-stack NVFP4 LLM pretraining is stable with a 1.47% loss gap to BF16 via LoRA-SVD projections, transformed AdamW/Root optimizers, and mixed-precision attention.
-
OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration
OASIS enables efficient LLM inference with non-uniform 4-bit weights and activations via precomputed Cartesian product lookup tables and a parallel outlier-compensation branch, at a reported 1.94-2.05% average accuracy drop.
-
Reference Traces for Auditing Invisible Weight Updates and Guiding Exact-Budget Protection
Weight updates that fall below half a ULP freeze coordinates deterministically, and freeze time is predictable a priori from a high-precision trajectory and mantissa length alone.
-
Stable FP4 Training via Transposition-Invariant Block Quantization
Transposition-invariant 2D block FP4 quantization plus truncation-free scaling and stochastic rounding enables stable end-to-end FP4 LLM training within ~1% of BF16.
-
CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits
CANN Bench is an Ascend-NPU benchmark with 53 operators and 1,060 released test cases that scores agent-generated kernels on compilation, correctness, and hardware-anchored performance.
-
What is New in Stochastic Rounding: a Survey on Theory, Hardware, and Applications
An update survey of stochastic rounding (2022–2026) that centers limited-precision SR, commercial hardware, probabilistic error bounds, and applications in ML and scientific computing.
Discussion (0). Sign in to comment.