Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

A hybrid-granularity FP8 recipe makes 8-bit training match BF16 accuracy on reasoning benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 14:48 UTC pith:NAOOC6B4

load-bearing objection A practical FP8 recipe worth refereeing, but the 'essentially lossless' claim is fully verified only at 1.5B; the 7B flagship lacks a matched BF16 continual pre-training baseline. the 3 major comments →

arxiv 2509.22536 v5 pith:NAOOC6B4 submitted 2025-09-26 cs.CL cs.AI

A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models

Wenjun Wang , Shuo Cai , Congkai Xie , Mingfa Feng , Yiming Zhang , Zhen Li , Kejing Yang , Ming Li
show 2 more authors
Jiannong Cao Hongxia Yang
This is my paper
classification cs.CL cs.AI
keywords FP8 traininglow-precision LLM traininghybrid granularity quantizationUE8M0 scaling factorcontinual pre-trainingsupervised fine-tuningreasoning benchmarkstraining efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that training large language models in FP8, not just BF16, can be a practical default: it claims an end-to-end recipe for continual pre-training and supervised fine-tuning that keeps model quality within evaluation noise of the BF16 baseline while cutting training time, memory, and compute. The recipe's core is hybrid-granularity quantization — block-wise scales for weights, token-wise scales for activations — combined with an exponent-only scaling-factor format that rounds up to powers of two. The paper validates the claim on a 160B-token continual pre-training run plus two-stage SFT, reporting nearly identical loss curves and benchmark scores within 1–2 points, alongside up to 22% faster training, 14% lower peak memory, and 19% higher throughput. If correct, the work turns FP8 from a promising research direction into a drop-in replacement that makes large-scale reasoning-model training cheaper and more accessible.

Core claim

On the paper's own terms, the central discovery is that FP8 training can be essentially lossless when quantization granularity is matched to tensor type: weights get block-wise quantization, activations get finer token-wise quantization, and the scaling factor uses an E8M0 exponent-only format that rounds upward to powers of two so the FP8 range is never overpushed. Throughout a 160B-token continual pre-training of a 1.5B model, FP8 and BF16 loss curves overlap almost exactly, and after supervised fine-tuning the models score within a 1–2 point band — attributed to evaluation noise — on reasoning benchmarks. The same recipe applied to a larger 7B model yields a reasoning model that substanti

What carries the argument

The load-bearing object is a hybrid-granularity quantization scheme that splits FP8 tensors into two regimes: weights are quantized per-block (small sub-matrices, each with its own scaling factor), while activations are quantized per-token (a distinct scale per token position), because activations have wider dynamic ranges and strong outliers. Scaling factors are stored in a UE8M0 (E8M0) format that represents only an exponent and rounds each scale up to the nearest power of two; this guarantees the quantized tensor stays within the FP8 range and stabilizes training. Master weights, weight gradients, and optimizer states remain in FP32, so small updates accumulate without rounding loss. The

Load-bearing premise

The losslessness claim rests on a 1.5B model where FP8 and BF16 were directly compared; if the same equivalence does not hold for the 7B model — which was trained only in FP8 — the central claim is unproven for the flagship result.

What would settle it

Train the identical 7B model and data mixture in BF16 under the same recipe and compare loss curves and benchmark scores with the published FP8 run; if the gap exceeds the 1–2 point evaluation-noise band, or the loss curves diverge, the essentially lossless claim fails. A cheaper check: re-run the 1.5B comparison with several seeds to see whether the observed FP8-vs-BF16 deltas are stable or just noise.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • FP8 can be used for the full training pipeline — continual pre-training and two-stage SFT — without a benchmark-quality penalty.
  • Training time drops by up to 22%, peak memory by up to 14%, and throughput rises by up to 19% relative to BF16.
  • The near-identical loss curves imply FP8 training preserves learning dynamics, not just final scores.
  • The recipe's efficiency gains allow larger batch sizes or longer contexts on the same hardware.
  • Models trained with this recipe on a 7B base show large reasoning gains over their starting checkpoint.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 1.5B head-to-head validates losslessness; the 7B model has no BF16 twin under the same recipe, so the strongest version of the claim — lossless at scale — remains an extrapolation.
  • The reported speedups (up to 22%) are well below FP8's theoretical 2x; the gap suggests quantization and communication overheads dominate, and a tuned kernel implementation could push efficiency higher.
  • FP8's occasional accuracy gains on the smaller model could come from quantization acting as a regularizer; a control experiment with multiple seeds would separate that effect from noise.
  • Memory savings would be most valuable in long-context or MoE training, where activation memory dominates; the recipe is a natural starting point for those regimes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents InfiR2, an end-to-end FP8 training recipe for continual pretraining and supervised fine-tuning of reasoning-oriented LLMs. The method combines block-wise weight quantization, token-wise activation quantization, E4M3 for forward/backward passes, UE8M0 power-of-two scaling factors, and FP32 master weights/gradients/optimizer states. Experiments include 160B-token continual pretraining of Qwen2.5-1.5B/7B models followed by two-stage SFT, plus a controlled BF16 vs FP8 comparison on the 1.5B model (Table 5, Figure 4) and SFT-only comparisons on Qwen2.5-Math models (Table 4). The paper reports training-time reductions up to 22%, peak-memory reductions up to 14%, and throughput gains up to 19% (Table 6), and claims FP8 training is 'essentially lossless' relative to BF16. The flagship artifact is InfiR2-7B-FP8, released with code and checkpoints.

Significance. If validated, the paper makes a practical contribution: a reusable, open FP8 recipe for large-scale continual pretraining and SFT, with a head-to-head 1.5B full-pipeline comparison showing overlapping loss curves and broadly comparable benchmark scores. The release of code, training logs, and checkpoints is a concrete strength that lowers the barrier to reproducing and extending FP8 training. The main scientific claim, however, is a losslessness claim, and that claim is currently supported directly only at the 1.5B scale; the 7B flagship lacks a BF16 counterpart under the same recipe. Given the field's reliance on benchmark deltas, the 2.46-point GPQA gap in the key validation table also needs statistical grounding. These issues do not invalidate the recipe, but they should be fixed before the paper's central claim can be accepted as stated.

major comments (3)
  1. [Section 5.3, Table 5] The central 'essentially lossless' claim rests on this single full-pipeline BF16 vs FP8 comparison at the 1.5B scale. On GPQA the FP8 model scores 29.48 versus 31.94 for BF16, a 2.46-point gap, which is larger than the '1-2 point margin attributable to evaluation noise' invoked in Sections 5.2.3 and 6. The paper provides no confidence intervals, standard errors, or multiple-seed runs for any benchmark, so the reader cannot tell whether this gap is noise or systematic. Please either report statistical uncertainty or soften the claim to 'typically within 1-2 points, with one exception' and discuss the direction of the discrepancy.
  2. [Section 5.1.2-5.1.3, Table 2, Figure 3] The flagship InfiR2-7B-FP8 has no BF16-trained counterpart under the same 160B-token continual-pretraining recipe. Figure 3 shows only FP8 loss curves at 7B, while Table 4 is an SFT-only comparison starting from Qwen2.5-Math models and does not exercise the accumulation of FP8 quantization errors over 160B tokens of continual pretraining. Thus the losslessness claim is unverified at the scale where the paper's main model sits. This is the key load-bearing gap. I recommend either providing a 7B BF16 run under the same recipe (even with a reduced token budget) or explicitly limiting the losslessness claim to the 1.5B scale and reframing the 7B results as a demonstration of recipe stability rather than losslessness.
  3. [Section 5.4, Table 6] The efficiency claims are reported as 'up to' maxima, and the underlying measurement conditions are incomplete. In two of the four configurations the FP8 forward pass is slower than BF16 (1.5B/32k: 875 vs 841 ms; 7B/8k: 2300 vs 1760 ms), and the peak-memory ratio at 7B/8k is only 0.95. The paper should state the hardware (GPU model), software stack (kernel library, CUDA version), number of repeats, and whether reported times are medians or means. Without this, the reader cannot assess how representative the headline 22%/14%/19% gains are, and the claim of a 'comprehensive recipe' is difficult to reproduce.
minor comments (6)
  1. [Tables 1 and 3] The continual-pretraining configuration lists only batch size, learning rates, weight decay, and context length; it omits optimizer, scheduler, warm-up, gradient clipping, and precision layout of optimizer states. The SFT table omits the optimizer and scheduler details as well. Please provide the full configuration for reproducibility.
  2. [Section 5.2.1] The variant 'FP8 w. FP32 scale' in Table 4 is not defined in the setup. Please clarify how it differs from the standard FP8 variant (e.g., whether UE8M0 scaling is replaced by ordinary FP32 scaling).
  3. [Figures 3 and 4] The term 'Decay' in the legend is not explained. I assume it denotes a learning-rate decay phase; please define it in the caption or text.
  4. [Table 6] The throughput ratio for the 1.5B/8k configuration is listed as 1.10x while the total-time ratio is 0.78x. The relationship between these metrics is not fully transparent; please state how throughput is computed and why it does not exactly track total time.
  5. [General / References] There are numerous typographical errors and malformed references: 'Minimun Learning Rate' (Table 1), 'performe' (Section 5.2.1), 'Daviditat B. de la Torre' and 'annd Fan-Kai Lin' in the Nemotron reference, and inconsistent URL formatting. A careful proofreading pass is needed.
  6. [Appendix A.1] The LLM usage statement is transparent and useful; I suggest including it in the main text or at least keeping it in the version of record.

Circularity Check

0 steps flagged

No significant circularity: FP8 losslessness is an empirical comparison against external benchmarks, with no fitted parameters or self-cited uniqueness theorems forcing the result.

full rationale

The paper's central claim is that FP8 training is essentially lossless relative to BF16. This claim is supported by paired training runs under an identical recipe, with the 1.5B model directly compared against BF16 in Section 5.3 (Figure 4 and Table 5), and by SFT comparisons on Qwen2.5-Math models in Section 5.2 (Table 4). The quantization strategy (block-wise weights, token-wise activations, UE8M0 scaling, FP32 master weights/gradients/optimizer states) is fixed before evaluation; no parameter is fitted to the benchmark scores used to assess losslessness. The evaluation uses external benchmarks (AIME24/25, GPQA, LiveCodeBench), not quantities that are defined in terms of the recipe's own outputs. Citations to prior work (DeepSeek-V3, COAT, UE8M0/MXFP8) supply techniques and hardware formats, but the losslessness conclusion is not derived from those citations; it is tested empirically. The InfiAlign datasets are authored by the same group, but they are used as training data for both FP8 and BF16 arms, so they do not bias the precision comparison. The absence of a BF16 twin for the flagship InfiR2-7B-FP8 under the full 160B-token continual-pretraining recipe is a generalization/validation gap, not a circularity: the 7B claim is an extrapolation, not a result forced by construction. Similarly, the GPQA gap of 2.46 points in Table 5 exceeds the paper's stated 1-2 point noise band, but that is a correctness or calibration issue, not a circularity. Overall, the derivation chain is self-contained against external benchmarks, and no definitional or fitted-input circularity is present.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The recipe uses standard quantization machinery with no fitted constants or new theoretical entities. The main unstated reliance is on the correctness of prior FP8 kernel implementations and on the representativeness of the benchmark suite for reasoning capability.

free parameters (1)
  • per-block quantization block size (bs)
    The block size for weight quantization is not reported; it is a manually chosen hyperparameter that trades accuracy against compute and affects the efficiency measurements.
axioms (3)
  • domain assumption BF16 training is a valid baseline and the benchmark suite captures the relevant differences between precisions.
    The paper relies on benchmarks (AIME, GPQA, LiveCodeBench) as proxies for reasoning capability, and treats BF16 as the reference standard without discussing its own limitations.
  • domain assumption UE8M0 scaling with power-of-two rounding preserves numerical fidelity as described in Mishra et al., and the FP8 GEMM kernels behave as assumed.
    The recipe depends on prior implementations (DeepGEMM, Mishra et al.) being correct and stable; errors there would invalidate the losslessness conclusion.
  • domain assumption Differences of 1-2 points between runs are 'inherent evaluation noise' without reported variance.
    The paper asserts that small gaps are noise, but no repeated runs or confidence intervals are provided, so this assumption is unverified.

pith-pipeline@v1.3.0-alltime-deepseek · 11307 in / 10034 out tokens · 90127 ms · 2026-08-04T14:48:54.515254+00:00 · methodology

0 comments
read the original abstract

The immense computational cost of training Large Language Models (LLMs) presents a major barrier to innovation. While FP8 training offers a promising solution with significant theoretical efficiency gains, its widespread adoption has been hindered by the lack of a comprehensive, open-source training recipe. To bridge this gap, we introduce an end-to-end FP8 training recipe that seamlessly integrates continual pre-training and supervised fine-tuning. Our methodology employs a fine-grained, hybrid-granularity quantization strategy to maintain numerical fidelity while maximizing computational efficiency. Through extensive experiments, including the continue pre-training of models on a 160B-token corpus, we demonstrate that our recipe is not only remarkably stable but also essentially lossless, achieving performance on par with the BF16 baseline across a suite of reasoning benchmarks. Crucially, this is achieved with substantial efficiency improvements, including up to a 22% reduction in training time, a 14% decrease in peak memory usage, and a 19% increase in throughput. Our results establish FP8 as a practical and robust alternative to BF16, and we will release the accompanying code to further democratize large-scale model training.

Figures

Figures reproduced from arXiv: 2509.22536 by Congkai Xie, Hongxia Yang, Jiannong Cao, Kejing Yang, Mingfa Feng, Ming Li, Shuo Cai, Wenjun Wang, Yiming Zhang, Zhen Li.

Figure 1
Figure 1. Figure 1: An illustration of three common quantization granularities: per-tensor, per-block, and per [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An illustration of a hybrid granularity quantization strategy using FP8, compared to a [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The FP8 training loss of InfiR2-1.5B and InfiR2-7B. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The validation loss and training loss of Continue Pretraining Qwen2.5-1.5B-base compar [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FeatCal: Feature Calibration for Post-Merging Models

    cs.LG 2026-05 conditional novelty 7.0

    FeatCal reduces feature drift in merged models via layer-wise closed-form calibration on a small dataset, outperforming prior post-merging methods on CLIP and GLUE benchmarks with high sample efficiency.

Reference graph

Works this paper leans on

45 extracted references · 25 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Am-deepseek-r1-0528-distilled, June 2025

    a-m team. Am-deepseek-r1-0528-distilled, June 2025. URL https://github.com/a-m-team/a-m-models

    2025

  2. [2]

    Understanding and overcoming the challenges of efficient transformer quantization

    Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint arXiv:2109.12948, 2021

    Pith/arXiv arXiv 2021

  3. [3]

    Quantizable transformers: Removing outliers by helping attention heads do nothing

    Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing. Advances in Neural Information Processing Systems, 36: 0 75067--75096, 2023

    2023

  4. [4]

    Infialign: A scalable and sample-efficient framework for aligning llms to enhance reasoning capabilities, 2025

    Shuo Cai, Su Lu, Qi Zhou, Kejing Yang, Zhijie Sang, Congkai Xie, and Hongxia Yang. Infialign: A scalable and sample-efficient framework for aligning llms to enhance reasoning capabilities, 2025. URL https://arxiv.org/abs/2508.05496

    Pith/arXiv arXiv 2025

  5. [5]

    Chuang, Yijun Liu, Pinjia He, Hong-Gyu Jung, and Lichao Sun

    Yufan Cai, Junjie Wang, Abhilasha Lodha, Yuxuan Liu, Shengyu Zhang, Ge Li, Yunfei Zhao, Jiazhen Gu, Haotian Cui, Zihan Wang, Chenyan Gu, Zhijian Wu, Zheyuan Zhang, Qingcheng Xiao, Yuxiang Wei, Jialun Wu, Xuanhe Zhou, Yichi Zhang, Terry Yue Zhuo, Zhiheng Xi, Wen-Ting S. Chuang, Yijun Liu, Pinjia He, Hong-Gyu Jung, and Lichao Sun. LiveCodeBench : A new benc...

    Pith/arXiv arXiv 2024

  6. [6]

    de la Torre, Deh-Rong Hung, Dhruti Shah, Dongji Feng, Dragomir Radev, Ed H

    Aakanksha Chowdhery, Abhishek Kadian, Adam Roberts, Adrien Le Scao, Aman Agarwal, Ani Nrusimha, Anmol Gulati, Anush Moorthy, Behnam Neyshabur, Biao Zhang, Bowen Yang, Chen Chen, Chris Alberti, Cindy Wang, Clemencia Siro, Daviditat B. de la Torre, Deh-Rong Hung, Dhruti Shah, Dongji Feng, Dragomir Radev, Ed H. Chi, Emanuele Bugliarello annd Fan-Kai Lin, Fan...

    2024

  7. [7]

    Deepgemm: clean and efficient fp8 gemm kernels with fine-grained scaling

    deepseek ai. Deepgemm: clean and efficient fp8 gemm kernels with fine-grained scaling. GitHub repository, 2024. URL https://github.com/deepseek-ai/DeepGEMM

    2024

  8. [8]

    Deepseek-v3 technical report, 2025

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, et al. Deepseek-v3 technical report, 2025. URL https://arxiv.org/abs/2412.19437

    Pith/arXiv arXiv 2025

  9. [9]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35: 0 30318--30332, 2022

    2022

  10. [10]

    Scaling fp8 training to trillion-token llms, 2025

    Maxim Fishman, Brian Chmiel, Ron Banner, and Daniel Soudry. Scaling fp8 training to trillion-token llms, 2025. URL https://arxiv.org/abs/2409.12517

    Pith/arXiv arXiv 2025

  11. [11]

    Binarized neural networks

    Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. Advances in neural information processing systems, 29, 2016

    2016

  12. [12]

    A study of bfloat16 for deep learning training, 2019

    Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. A study of bfloat16 for de...

    Pith/arXiv arXiv 2019

  13. [13]

    The stack v2: The code moe of bigcode, 2023

    Denis Kocetkov, Harm de Vries, Arjun Ashok, Maxim Borisyak, Qian Liu, Chenghao Mou, Danish Contractor, Tri Dao, Raymond Li, Yacine Jernite, Sean Hughes, Thomas Wolf, Leandro von Werra, and Niklas Muennighoff. The stack v2: The code moe of bigcode, 2023

    2023

  14. [14]

    Bert busters: Outlier dimensions that disrupt transformers

    Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. Bert busters: Outlier dimensions that disrupt transformers. arXiv preprint arXiv:2105.06990, 2021

    Pith/arXiv arXiv 2021

  15. [15]

    To fp8 and back again: Quantifying the effects of reducing precision on llm training stability

    Joonhyung Lee, Jeongin Bae, Byeongwook Kim, Se Jung Kwon, and Dongsoo Lee. To fp8 and back again: Quantifying the effects of reducing precision on llm training stability. CoRR, 2024

    2024

  16. [16]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems, 6: 0 87--100, 2024

    2024

  17. [17]

    Starcoder 2 and the stack v2: The next generation

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024

    Pith/arXiv arXiv 2024

  18. [19]

    Mixed precision training, 2018

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training, 2018. URL https://arxiv.org/abs/1710.03740

    Pith/arXiv arXiv 2018

  19. [20]

    Fp8 formats for deep learning, 2022

    Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart Oberman, Mohammad Shoeybi, Michael Siu, and Hao Wu. Fp8 formats for deep learning, 2022. URL https://arxiv.org/abs/2209.05433

    Pith/arXiv arXiv 2022

  20. [21]

    Recipes for pre-training llms with mxfp8, 2025

    Asit Mishra, Dusan Stosic, Simon Layton, and Paulius Micikevicius. Recipes for pre-training llms with mxfp8, 2025. URL https://arxiv.org/abs/2506.08027

    Pith/arXiv arXiv 2025

  21. [22]

    Evalscope

    ModelScope. Evalscope. URL https://github.com/modelscope/evalscope

  22. [23]

    Nvidia blackwell architecture technical brief

    NVIDIA . Nvidia blackwell architecture technical brief. Technical report, NVIDIA, 2024. URL: https://resources.nvidia.com/en-us-blackwell-architecture

    2024

  23. [24]

    Gpt-4 technical report, 2024

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    Pith/arXiv arXiv 2024

  24. [25]

    The fineweb dataset

    Guillermo Penedo, Angela Fan, Dan Hesslow, Ruxandra Cojocaru, Sasha Luccioni, Irene Alistar, Julie Fromken, Baptiste Pannier, Teven Villanova, and Teven Le Scao. The fineweb dataset. https://huggingface.co/datasets/HuggingFaceFW/fineweb, 2024

    2024

  25. [26]

    Fp8-lm: Training fp8 large language models, 2023

    Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, Ruihang Li, Miaosen Zhang, Chen Li, Jia Ning, Ruizhe Wang, Zheng Zhang, Shuguang Liu, Joe Chau, Han Hu, and Peng Cheng. Fp8-lm: Training fp8 large language models, 2023. URL https://arxiv.org/abs/2310.18313

    Pith/arXiv arXiv 2023

  26. [27]

    Qwen2.5 technical report, 2025

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115

    Pith/arXiv arXiv 2025

  27. [28]

    Marblestone, and Percy Liang

    David Rein, Stas Gaskin, Luyu Gao, John Miller, Pang Wei Koh, Jackson Kernion, Adam H. Marblestone, and Percy Liang. GPQA : A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. URL https://arxiv.org/abs/2311.12022

    Pith/arXiv arXiv 2023

  28. [29]

    Ocp microscaling (mx) specification

    Bita Darvish Rouhani, Nitin Garegrat, Tom Savell, Ankit More, Kyung-Nam Han, Ritchie Zhao, Mathew Hall, Jasmine Klar, Eric Chung, Yuan Yu, Michael Schulte, Ralph Wittig, Ian Bratt, Nigel Stephens, Jelena Milanovic, John Brothers, Pradeep Dubey, Marius Cornea, Alexander Heinecke, Andres Rodriguez, Martin Langhammer, Summer Deng, Maxim Naumov, Paulius Micik...

    2023

  29. [30]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL https://arxiv.org/abs/2403.05530

    Pith/arXiv arXiv 2024

  30. [31]

    Not all correct answers are equal: Why your distillation source matters, 2025

    Xiaoyu Tian, Yunjie Ji, Haotian Wang, Shuaiting Chen, Sitong Zhao, Yiping Peng, Han Zhao, and Xiangang Li. Not all correct answers are equal: Why your distillation source matters, 2025. URL https://arxiv.org/abs/2505.14464

    Pith/arXiv arXiv 2025

  31. [32]

    Le, He He, and Galyna Vasylyeva

    Thang Luong Trinh, Yuhuai Wu, Quoc V. Le, He He, and Galyna Vasylyeva. The aimo prize: A new benchmark for advanced mathematical reasoning. arXiv preprint arXiv:2407.00532, 2024. URL https://arxiv.org/abs/2407.00532

    Pith/arXiv arXiv 2024

  32. [33]

    Gopalakrishnan

    Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and K. Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. ArXiv, abs/1812.08011, 2018 a . URL https://api.semanticscholar.org/CorpusID:53977760

    Pith/arXiv arXiv 2018

  33. [34]

    Training deep neural networks with 8-bit floating point numbers

    Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. Advances in neural information processing systems, 31, 2018 b

    2018

  34. [35]

    Stable and low-precision training for large-scale vision-language models, 2023

    Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low-precision training for large-scale vision-language models, 2023. URL https://arxiv.org/abs/2304.13013

    Pith/arXiv arXiv 2023

  35. [36]

    Training and inference with integers in deep neural networks

    Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. arXiv preprint arXiv:1802.04680, 2018

    Pith/arXiv arXiv 2018

  36. [37]

    Training transformers with 4-bit integers, 2023

    Haocheng Xi, Changhao Li, Jianfei Chen, and Jun Zhu. Training transformers with 4-bit integers, 2023. URL https://arxiv.org/abs/2306.11987

    Pith/arXiv arXiv 2023

  37. [38]

    Jetfire: Efficient and accurate transformer pretraining with INT 8 data flow and per-block quantization

    Haocheng Xi, Yuxiang Chen, Kang Zhao, Kai Jun Teh, Jianfei Chen, and Jun Zhu. Jetfire: Efficient and accurate transformer pretraining with INT 8 data flow and per-block quantization. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Con...

    2024

  38. [39]

    Coat: Compressing optimizer states and activation for memory-efficient fp8 training, 2025

    Haocheng Xi, Han Cai, Ligeng Zhu, Yao Lu, Kurt Keutzer, Jianfei Chen, and Song Han. Coat: Compressing optimizer states and activation for memory-efficient fp8 training, 2025. URL https://arxiv.org/abs/2410.19313

    Pith/arXiv arXiv 2025

  39. [40]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pp.\ 38087--38099. PMLR, 2023

    2023

  40. [41]

    Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients

    Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160, 2016

    Pith/arXiv arXiv 2016

  41. [42]

    Towards unified int8 training for convolutional neural network, 2019

    Feng Zhu, Ruihao Gong, Fengwei Yu, Xianglong Liu, Yanfei Wang, Zhelong Li, Xiuqi Yang, and Junjie Yan. Towards unified int8 training for convolutional neural network, 2019. URL https://arxiv.org/abs/1912.12607

    Pith/arXiv arXiv 2019

  42. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  43. [44]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  44. [45]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  45. [46]

    Z5R[zz PvԿ !1 y ,w 8

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 1999