Pith. sign in

REVIEW 3 major objections 4 minor 9 cited by

TorchAO: PyTorch-Native Training-to-Serving Model Optimization

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TorchAO unifies LLM optimization from FP8 training to quantized serving in one PyTorch-native workflow.

desk verdict A genuinely useful open-source quantization/sparsity stack, but the paper's headline end-to-end claim is only component-benchmarked, not demonstrated as a full pipeline. read the letter →

arxiv 2507.16099 v1 pith:ZB3JCTAX submitted 2025-07-21 cs.LG

Andrew Or , Apurva Jain , Daniel Vega-Myhre , Jesse Cai , Charles David Hernandez , Zhenrui Zheng , Driss Guessous , Vasiliy Kuznetsov
show 5 more authors
Christian Puhrsch Mark Saroufim Supriya Rao Thien Tran Aleksandar Samardžić
This is my paper
classification cs.LG
keywords modeloptimizationquantizationFP8trainingquantization-awarepost-trainingsparsitytensorsubclassLLMserving
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TorchAO aims to close the gap between training and serving for large language models by providing a single, PyTorch-native optimization framework that covers FP8 quantized pre-training, quantization-aware fine-tuning (QAT), post-training quantization (PTQ), and 2:4 sparsity. The paper argues that a unified workflow built on PyTorch's tensor subclass abstraction can represent many low-precision data types (INT4, INT8, FP8, MXFP4, MXFP6, MXFP8) in a backend-agnostic way, so the same quantized model can move from TorchTitan pre-training, to TorchTune or Axolotl fine-tuning, to vLLM, SGLang, HuggingFace, and ExecuTorch serving without format conversion or numeric drift. The payoff, if true, is that quantization becomes a first-class, end-to-end decision rather than a fragmented set of per-step hacks, and the paper reports concrete gains: 1.25x FP8 training speedup on Llama3-8B (up to 1.5x at 405B scale), up to 2x serving throughput from PTQ with 2-4x smaller models, and QAT recovering up to 69.8% of hellaswag accuracy degradation and 82.8% of wikitext perplexity degradation. The same QAT workflow underpins the shipped quantized Llama 3.2 1B/3B and LlamaGuard3-8B checkpoints.

What carries the argument

The tensor subclass abstraction is the central object: a PyTorch mechanism that lets a custom low-precision data type (INT4, INT8, FP8, MXFP4, MXFP6, MXFP8) be represented as a subclass of torch.Tensor, carrying its own quantized storage and kernel dispatch while still composing with autograd, torch.compile, FSDP2, tensor parallelism, and serialization. Around this core, the framework contributes three recipe families: FP8 training scaling recipes (tensorwise, rowwise, and rowwise with high-precision gradient-weight) that cast activations, weights, and gradients to FP8 for tensor-core GEMMs; a two-step QAT flow (prepare inserts fake-quantization ops, convert swaps them for real quantized ops through the same PTQ code path); and sparsity kernels including 2:4, block, and sparse Marlin layouts. These pieces do the work of keeping numerics consistent from training to serving and of lowering quantized models to CUDA, ARM CPU, Metal, and XNNPACK backends.

What would settle it

Re-run the Llama3-8B FP8 training benchmark (8x H100, batch size 1, sequence length 8192, 100 steps, tensorwise scaling with FP8 all-gather) across multiple seeds and hardware revisions and compare the throughput distribution against the 1.25x claim; if the variance spans the gap to 1.0x, the quantitative claim does not generalize. Similarly, re-running the Llama3.1-8B QAT fine-tuning for 1000 steps with several seeds would test whether the 69.8%-82.8% recovery figures reproduce.

Watch

Extended reading notes

Core claim

The central claim is that a single PyTorch-native framework can host the full model-optimization lifecycle—FP8 training, QAT, PTQ, and 2:4 sparsity—without sacrificing composability with autograd, torch.compile, FSDP2, tensor parallelism, or serialization. The load-bearing design is the tensor subclass abstraction: low-precision numerics are packaged as torch.Tensor subclasses so that quantization and sparsity behave like ordinary tensors throughout the stack. TorchAO states that this yields measurable results: tensorwise FP8 training on Llama3-8B reaches 1.25x throughput over BF16 with on-par peak memory, PTQ on Llama3.1-8B produces 2-4x smaller models with up to 2x inference throughput at near-parity accuracy, and QAT on Llama3-8B/3.1-8B/3.2-3B recovers roughly 58-70% of accuracy degradation and 78-83% of perplexity degradation at INT4. The paper further claims the QAT workflow was used in the released quantized Llama 3.2 1B/3B and LlamaGuard3-8B models, giving 2-4x inference speedup, 56% size reduction, and 41% memory reduction versus BF16.

Load-bearing premise

The reported speedups and recovery percentages come from single runs on a narrow configuration set—batch size 1 for PTQ and serving, batch size 8 over 1000 steps for QAT, and 100 steps for FP8 training—so the paper assumes these numbers are stable enough to generalize across seeds, hardware, and longer training runs.

Editorial extensions

If this is right

  • The same FP8 configuration used during training can be used during serving in vLLM, so models do not need format conversion or re-quantization between steps.
  • Fine-tuning with TorchAO QAT, optionally composed with LoRA at 1.89x throughput over vanilla QAT, can recover most INT4 accuracy loss, making 4-bit mobile deployment practical.
  • PTQ on a Llama3.1-8B-class model can cut model size by 2-4x and roughly double inference throughput with near-parity hellaswag and wikitext scores.
  • Because quantization and sparsity are expressed as tensor subclasses, the same optimized model can be serialized with HuggingFace APIs, served by vLLM or SGLang, and lowered to ExecuTorch for Android and iOS.
  • The launch of quantized Llama 3.2 1B/3B and LlamaGuard3-8B demonstrates the workflow at production scale, with 2-4x speedup, 56% smaller models, and 41% lower memory.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if FP8 training and FP8 serving share scale recipes, quantization becomes a training-time decision rather than a post-hoc compression step, which may change how model checkpoints are released and reused.
  • Beyond the paper: because the benchmark uses batch size 1 for serving, the reported 2x throughput likely understates the advantage at larger batches where compute-bound GEMMs benefit more from low precision.
  • Beyond the paper: the tensor-subclass design is a testable general mechanism—one could implement a new low-precision format purely as a subclass and check whether it composes with autograd, torch.compile, and distributed training without kernel changes.
  • Beyond the paper: the 100-step and 1000-step training runs leave open whether the speedups and QAT recovery persist to convergence on a full-scale training run; running FP8 training to completion on a smaller model would settle this directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents TorchAO, an open-source PyTorch-native model optimization framework that supports FP8 quantized training, quantization-aware training (QAT), post-training quantization (PTQ), and 2:4 sparsity through a tensor-subclass abstraction. It describes two intended workflows: FP8 training/fine-tuning served on GPUs via vLLM, and QAT fine-tuning lowered to mobile via ExecuTorch/XNNPACK. The evaluation reports component-level benchmarks: FP8 pretraining on Llama3-8B (Table 3), PTQ on Llama3.1-8B (Table 4), QAT on Llama 3/3.1/3.2 (Table 2), and FP8 serving on Llama3.1-8B (Table 1).

Significance. If the claims hold, TorchAO is a practically useful integration layer that reduces format-conversion and numerics-mismatch friction across the training-to-serving lifecycle. The paper's strengths are its public codebase, the concrete API and command-line listings, and the clear description of FP8 scaling recipes in Appendix A. The tables are broadly internally consistent, apart from a few rounding discrepancies. The principal weakness is that the evaluation is component-wise rather than end-to-end, so the headline integration claim is not directly tested.

major comments (3)
  1. [§4 Evaluation; §2.3 Serving] The central end-to-end claim is not evaluated as such. The component benchmarks use different models and configurations: FP8 training on Llama3-8B for 100 steps (Table 3), PTQ on Llama3.1-8B at batch size 1 (Table 4), QAT on Llama3/3.1/3.2 (Table 2), and serving on Llama3.1-8B with one prompt (Table 1). No experiment trains or fine-tunes a model with TorchAO and then serves the same checkpoint with the same quantization configuration. The assertion in §2.3 that FP8 inference uses the same configurations as FP8 training to provide consistent end-to-end numerics is therefore unsupported by the presented evidence. Please add an explicit end-to-end run, or explicitly scope the claim to component-wise integration and state that end-to-end numerical consistency is asserted but not benchmarked.
  2. [§4 Evaluation; Tables 1-4] The quantitative performance claims rest on single runs without error bars, multiple seeds, or significance tests. Training runs last 100 steps in Table 3 and 1000 steps in Table 2, PTQ uses a batch size of 1 in Table 4, and the serving benchmark in Table 1 uses a single prompt. These conditions are too narrow to support general statements such as a 1.25x training speedup, a 28% throughput increase, and up to 2x inference throughput. Report repeated runs with variance or state explicitly that the numbers are indicative single-sample measurements.
  3. [§1 bullet list; Table 2] The paper claims in §1 that QAT can recover up to 96% of the degradation in quantized accuracy, but Table 2 reports maximum recoveries of 69.8% for hellaswag accuracy and 82.8% for wikitext perplexity. The 96% figure comes from a self-cited blog post (Or et al., 2024) and is not reproduced in this evaluation. Please state the provenance of each recovery figure and avoid presenting external results as if they were part of the present benchmark.
minor comments (4)
  1. [Table 2] In the Llama3-8B row, the hellaswag recovery of 57.8% does not match the values in the row; recomputing (52.8-47.0)/(57.1-47.0) gives 57.4%, so the percentage appears to be a typo.
  2. [Table 1 caption] The caption states that clients used the ShareGPT dataset and 'number of prompts = 1'; a one-prompt serving benchmark is not representative of the multi-user workloads the framework targets, and the caption should clarify the protocol or acknowledge the limitation.
  3. [§2.1 and §3.1] The 1.5x 405B FP8 training speedup and the 1.89x QAT+LoRA throughput improvement are cited from non-archival blog posts; mark these explicitly as external results rather than measurements from this paper.
  4. [Appendix C] Figure 3 is referenced but the text does not explain how to interpret the speedup table, including the definitions of the M, N, K axes and any hardware configuration.

Circularity Check

2 steps flagged · score 2.0 of 10

No circular derivation: every quantitative claim is a benchmark measurement, and Section 4 re-measures each mechanism independently. The only circularity-adjacent feature is reliance on same-team blog posts (Wright et al. 2024; Or 2024; Or et al.

  1. self citation load bearing [Section 1 (Introduction), 'Fine-tuning' bullet; Section 2.1, FP8 Training]
    "TorchAO's QAT support can recover up to 96% of the degradation in quantized accuracy (Or et al., 2024) and can be composed with LoRA (Hu et al., 2022) to improve the training throughput by 1.89x compared to vanilla QAT (Or, 2024; MetaAI, 2025b)."

    The headline benefit figures are supported only by Or et al. (2024), a pytorch.org blog written by the present authors, and Or (2024), a dev-discuss post by the first author. The paper does not reproduce either number: its own Table 2 measures at most 69.8% recovery (hellaswag) and 82.8% (perplexity), and its own Table 3 measures 1.25x FP8 training speedup. The argumentative support for the largest benefit claims therefore reduces to the same team's earlier self-reports. This is self-citation in load-bearing position, but it is minor in weight: the cited posts contain real measurements of the same code, and Section 4 independently re-measures the same mechanisms at smaller scale, so the central framework claim does not collapse to the citations.

  2. self citation load bearing [Section 3.1, Quantization-Aware Training]
    "Recent launches of the quantized Llama 3.2 1B/3B (MetaAI, 2025b) and LlamaGuard3-8B models (Inan et al., 2023) leveraged TorchAO's QAT support to mitigate quantization degradation in their INT4 checkpoints targeting the ARM CPU backend. This resulted in a 2-4x inference speedup, 56% reduction in model size, and 41% reduction in memory usage compared to the original BF16 checkpoints."

    The paper's external-validation claim, that TorchAO enabled the launched quantized Llama models, plus the quoted speedup, size, and memory figures, is cited to MetaAI (2025b), a first-party Meta blog post, and to Inan et al. (2023), the original LlamaGuard paper that predates TorchAO. No experiment in this paper verifies that the released INT4 checkpoints were produced through TorchAO's QAT path, nor measures the quoted 2-4x, 56%, and 41% figures on those checkpoints. The launch-credit claim therefore rests on the authors' own organization's account. It is attestation rather than in-paper evidence and is not a definitional reduction; the released checkpoints are public artifacts, so the claim remains externally checkable.

full rationale

This is a systems and benchmarking paper with no derivation chain, so the classic circularity patterns (a quantity defined in terms of the quantity it is said to predict; a fitted parameter renamed as a prediction; an ansatz or uniqueness theorem imported from the authors' prior work) do not occur. Each quantitative claim is a measured outcome: Table 3 measures FP8 training throughput on Llama3-8B over 100 steps, Table 4 measures PTQ accuracy and throughput on Llama3.1-8B, and Table 2 measures QAT accuracy on three Llama models fine-tuned on OASST1. The QAT 'recovery' statistic is computed from three measured quantities (QAT-quantized accuracy, PTQ-quantized accuracy, and BF16 accuracy), so it is an outcome metric, not an input fitted to produce the claim. The same holds for the speedups and throughput numbers, which are raw measurements with no fitted parameters. I also weighed the paper's assertion in Section 2.3 that 'FP8 inference support uses the same configurations as FP8 training to provide consistent end-to-end numerics': this is a design claim that is asserted but never exercised by an integrated experiment (no section trains or fine-tunes a model with TorchAO and then serves that same checkpoint). That is an evaluation-coverage gap, not circularity: no equation or fitted value makes the consistency claim true by construction, and per the rules absence of evidence is a correctness risk, not a circular step. The genuinely noteworthy pattern is the density of same-team self-citation for headline numbers: the 1.5x speedup at 405B scale (Wright et al., 2024), the 0.96 recovery figure (Or et al., 2024), the 1.89x QAT+LoRA speedup (Or, 2024), the 2-4x edge speedup and launch credit (MetaAI, 2025b), and the 17% async-TP gain (torchtitan, 2025) are all attributed to blog posts and releases produced by the same group that wrote this paper. However, these cited items are real external measurements of public artifacts (a 512-GPU training run, released HuggingFace checkpoints) and are in principle checkable outside this paper's fitted values, while Section 4 anchors the same mechanisms with independent, smaller-scale measurements. The central claim therefore retains independent content, and the honest finding is minor self-citation (score 2), not circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No mathematical parameters or invented entities; the paper is an empirical systems report. The ledger contains only domain and methodological assumptions about hardware behavior, task representativeness, and benchmark stability.

assumptions (3)
  • domain assumption FP8 tensor cores on H100 GPUs provide significant GEMM speedups with negligible accuracy loss when using appropriate scaling recipes.
    The central training and serving speedup claims in Section 2.1, Section 2.3, and Table 3 rely on this hardware behavior.
  • domain assumption Hellaswag accuracy and Wikitext word perplexity are representative proxies for quantized model quality.
    Tables 2 and 4 use these two tasks to measure quantization degradation and QAT recovery; the claims about quality recovery are scoped to these tasks.
  • ad hoc to paper Benchmark runs without seeds, repeated trials, or significance testing reliably represent typical performance.
    Section 4 reports single-configuration numbers (batch size 8 for QAT, batch size 1 for PTQ) with no variance or significance testing, so the numerical claims rest on this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TorchAO: PyTorch-Native Training-to-Serving Model Optimization." pith.science (2026). https://pith.science/paper/ZB3JCTAX

@misc{pith2026250716099,
  author       = {Pith},
  title        = {Pith review of: TorchAO: PyTorch-Native Training-to-Serving Model Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZB3JCTAX}},
  note         = {Machine review of arXiv:2507.16099}
}
read the original abstract

We present TorchAO, a PyTorch-native model optimization framework leveraging quantization and sparsity to provide an end-to-end, training-to-serving workflow for AI models. TorchAO supports a variety of popular model optimization techniques, including FP8 quantized training, quantization-aware training (QAT), post-training quantization (PTQ), and 2:4 sparsity, and leverages a novel tensor subclass abstraction to represent a variety of widely-used, backend agnostic low precision data types, including INT4, INT8, FP8, MXFP4, MXFP6, and MXFP8. TorchAO integrates closely with the broader ecosystem at each step of the model optimization pipeline, from pre-training (TorchTitan) to fine-tuning (TorchTune, Axolotl) to serving (HuggingFace, vLLM, SGLang, ExecuTorch), connecting an otherwise fragmented space in a single, unified workflow. TorchAO has enabled recent launches of the quantized Llama 3.2 1B/3B and LlamaGuard3-8B models and is open-source at https://github.com/pytorch/ao/.

Figures

Figures reproduced from arXiv: 2507.16099 by the authors.

Figure 1
Figure 1. TorchAO optimization workflow. TorchAO is closely integrated with popular pre-training, fine-tuning, serving, and model-definition frameworks to provide a seamless, PyTorch￾native end-to-end workflow for users to optimize their models. ∗MX training is a prototype feature. Qwen3 (235B parameters) (Qwen3, 2025), DeepSeek-v3 (671B) (Guo et al., 2025), Llama 3.1 (405B) (Grattafiori et al., 2024), and Llama 4 Behemoth (2… view at source ↗
Figure 2
Figure 2. TorchAO APIs. Users can optimize their models using the above one-line transformations. ∗API subject to change. We present TorchAO, a PyTorch-native model optimization framework leveraging quantization and sparsity to provide an end-to-end, training-to-serving workflow for AI models. TorchAO integrates closely with the broader ecosystem at each step of the model optimization pipeline ( [PITH_FULL_IMAGE:figures/full… view at source ↗
Figure 3
Figure 3. Observed float8 vs bfloat16 speedup of LayerNorm + Linear + Sigmoid forward and backward pass, by forward M, N, K A common question about FP8 training is ”when is FP8 linear faster vs BF16?”. Given the M, K, N of the forward pass through your linear, you can reference the table below for a microbenchmark based speedup estimate on NVIDIA H100 10 [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Observed loss curves for float8 tensorwise and rowwise training with FSDP2 on 8xH100 GPUs, compared with the bfloat16 baseline. E. Appendix: TorchAO Prototype Features TorchAO currently has many exciting prototype features that are not guaranteed to be suitable for pro…

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Zero-Shot Quantization via Weight-Space Arithmetic

    cs.CV 2026-04 unverdicted novelty 8.0 of 10

    A quantization vector derived from a donor model via weight-space arithmetic can be added to a receiver model to improve post-PTQ Top-1 accuracy by up to 60 points in 3-bit settings without receiver-side QAT or data.

  2. StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Near-memory DeQuantization Blocks in the HBM base die dequantize weight-only quantized LLM weights on the load path, cutting GPU dequant overhead and yielding up to 7.08× mpGEMM speedup and 2.20× decode throughput in ...

  3. LoKA: Low-precision Kernel Applications for Recommendation Models At Scale

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    LoKA enables practical FP8 use in numerically sensitive large recommendation models via profiling, model adaptations, and runtime kernel orchestration.

  4. LoKA: Low-precision Kernel Applications for Recommendation Models At Scale

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    LoKA enables practical FP8 use in numerically sensitive large recommendation models via online profiling of activations, reusable model modifications for stability, and dynamic kernel dispatching.

  5. StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    StoSignSGD resolves SignSGD divergence on non-smooth objectives via structural stochasticity, matching optimal convex rates and improving non-convex bounds while delivering 1.44-2.14x speedups in FP8 LLM pretraining.

  6. torchtune: PyTorch native post-training library

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    torchtune is a modular PyTorch library for LLM post-training that delivers competitive performance and memory efficiency while supporting rapid research iteration through hackable components.

  7. LoKA: Low-precision Kernel Applications for Recommendation Models At Scale

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    LoKA claims to make FP8 practical for large recommendation models via statistical probing, model adaptations, and accuracy-aware kernel dispatch.

  8. Recti-Q: Feature-Space Rectification for Out-of-Distribution-Robust Quantized Perception in Edge Robotics

    cs.CV 2026-07 conditional novelty 4.0 of 10

    Recti-Q measures a 'Quantization-Induced Robustness Gap' in 4-bit PTQ vision models and shows a small head-level LoRA adapter trained on source data recovers part of the lost out-of-distribution accuracy.

  9. CAT-Translate: Building Compact Open-Source Models for Japanese-English Translation

    cs.CL 2026-06 unverdicted novelty 3.0 of 10

    Compact 0.8B-7B models for bidirectional Japanese-English translation outperform large multilingual models on real-world domain benchmarks.

Reference graph

Works this paper leans on

15 extracted references · 1 canonical work pages · cited by 7 Pith papers

  1. [4]

    Accelerating transformer inference and training with 2:4 activation sparsity

    Haziza, D., Chou, T., Choudhary, D., Wehrstedt, L., Massa, F., Yu, J., Jeong, G., Rao, S., Labatut, P., and Cai, J. Accelerating transformer inference and training with 2:4 activation sparsity. arXiv preprint arXiv:2503.16672 ,

    arXiv

  2. [6]

    Parq: Piecewise-affine regularized quantization

    Jin, L., Ma, J., Liu, Z., Gromov, A., Defazio, A., and Xiao, L. Parq: Piecewise-affine regularized quantization. arXiv preprint arXiv:2503.15748,

    arXiv

  3. [7]

    Torchtitan: One-stop pytorch native solution for production ready llm pre-training

    Liang, W., Liu, T., Wright, L., Constable, W., Gu, A., Huang, C.-C., Zhang, I., Feng, W., Huang, H., Wang, J., et al. Torchtitan: One-stop pytorch native solution for production ready llm pre-training. arXiv preprint arXiv:2410.06511,

    arXiv

  4. [8]

    Spinquant: Llm quantization with learned rotations

    Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V ., Tian, Y ., and Blankevoort, T. Spinquant: Llm quantization with learned rotations. arXiv preprint arXiv:2405.16406,

    arXiv

  5. [9]

    Paretoq: Scaling laws in extremely low-bit llm quantization

    Liu, Z., Zhao, C., Huang, H., Chen, S., Zhang, J., Zhao, J., Roy, S., Jin, L., Xiong, Y ., Shi, Y ., et al. Paretoq: Scaling laws in extremely low-bit llm quantization. arXiv preprint arXiv:2502.02631,

    arXiv

  6. [10]

    The llama 4 herd: The beginning of a new era of natively multimodal ai in- novation

    MetaAI. The llama 4 herd: The beginning of a new era of natively multimodal ai in- novation. https://ai.meta.com/blog/ llama-4-multimodal-intelligence/ , 2025a. MetaAI. Introducing quantized llama models with increased speed and a reduced memory footprint. https://ai.meta.com/blog/ meta-llama-quantized-lightweight-models/ , 2025b. Mishra, A., Latorre, J. ...

    arXiv

  7. [11]

    D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., et al

    Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537,

    arXiv

  8. [13]

    Diffusers: State-of-the- art diffusion models

    von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lambert, N., Rasul, K., Davaadorj, M., Nair, D., Paul, S., Liu, S., Berman, W., Xu, Y ., and Wolf, T. Diffusers: State-of-the- art diffusion models. URL https://github.com/ huggingface/diffusers. Wang, Y ., He, H., and Wehrstedt, L. Py- torch symmetricmemory: Harnessing nvlink programmability with ease. ...

    arXiv 1910

Show all 15 references
  1. [14]

    Galore: Memory-efficient llm train- ing by gradient low-rank projection

    Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y . Galore: Memory-efficient llm train- ing by gradient low-rank projection. arXiv preprint arXiv:2403.03507,

    arXiv

  2. [15]

    H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J

    Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. Sglang: Efficient execution of structured language model programs. https://arxiv.org/abs/2312.07104. 8 TorchAO: PyTorch-native Training-to-Serving Model Opti...

    arXiv

  3. [2021]

    Torchao: Low-bit arm cpu and metal ker- nels for linear and embedding ops

    torchao. Torchao: Low-bit arm cpu and metal ker- nels for linear and embedding ops. https: //github.com/pytorch/ao/tree/main/ torchao/experimental, 2025a. torchao. Torchao prototypes. https://github. com/pytorch/ao/tree/main/torchao/ prototype, 2025b. torchao. Torchao post-tra...

    2025

  4. [2022]

    Llama guard: Llm-based input-output safeguard for human-ai conversations

    Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y ., Tontchev, M., Hu, Q., Fuller, B., Testug- gine, D., et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674,

    arXiv

  5. [2023]

    Llm.int8(): 8-bit matrix multiplication for transformers at scale

    Dettmers, T., Lewis, M., Belkada, Y ., and Zettlemoyer, L. Llm.int8(): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339,

    arXiv

  6. [2024]

    Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

  7. [2025]

    The llama 3 herd of models

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.