Pith. sign in

REVIEW 2 major objections 4 minor 2 cited by

Speeding up Model Loading with fastsafetensors

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A library called fastsafetensors loads large language models 4.8x to 7.5x faster than the standard safetensors deserializer by copying groups of model parameters straight into GPU memory and instantiating tensors there with DLPack instead…

desk verdict Useful systems paper with real speedups, but the authors need to prove the loaded tensors are actually correct before the headline number means anything. read the letter →

arxiv 2505.23072 v1 pith:CV7QSPL5 submitted 2025-05-29 cs.DC

Takeshi Yoshimura , Tatsuhiro Chiba , Manish Sethi , Daniel Waddington , Swaminathan Sundararaman This is my paper
classification cs.DC
keywords modelloadingsafetensorstensordeserializationpeer-to-peerDMAGPUoffloadinglargelanguagemodelsinferenceserverstartupdatatransferoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fastsafetensors targets a specific inefficiency it identifies in the standard safetensors loader: parameters are deserialized one by one into host-memory tensor objects and then copied to the GPU, which leaves NVMe storage underutilized and makes model loading account for an average of 92 percent of inference-server startup latency in the paper's measurements. The library instead copies large contiguous groups of on-disk parameters straight into GPU memory and only then wraps those buffers as tensor objects using DLPack. This reordering enables parallelized reads, NUMA-aware placement, and peer-to-peer DMA where available, and it lets tensor-parallel sharding run on the GPU through collective broadcast and scatter. The paper reports 4.8x to 7.5x faster loading for Llama-7B/13B/70B, Falcon-40B, and Bloom-176B, bringing model loading from minute-scale to second-scale.

What carries the argument

The central mechanism is aggregated tensor deserialization: parsing the safetensors header to learn every tensor's offset, shape, and dtype, then copying whole groups of tensors as one contiguous device-memory buffer before any tensor object exists. Tensor objects are then created with DLPack, a protocol that wraps a raw pointer, device, data type, and strides as a framework tensor, which removes the per-tensor host-memory round trip. A second mechanism, called shuffling, reads files onto a round-robin set of GPUs and then uses collective broadcast and scatter to redistribute shards for tensor parallelism. Because safetensors does not guarantee alignment, the implementation adds a device-side bounce-buffer pass that fixes misaligned offsets after peer-to-peer DMA transfers and also hosts dtype conversions.

What would settle it

Load the same checkpoint and input through an inference server using the original safetensors loader and then through the same server using fastsafetensors, and compare every loaded weight and the full output tensors element-wise; a mismatch, crash, or misalignment error on a file with an odd-sized header would refute the claimed equivalence of the DLPack-wrapped tensors.

Watch

Extended reading notes

Core claim

Fastsafetensors establishes that tensor deserialization can be decoupled from tensor object creation. Using the safetensors header, the loader treats the file body as a set of known contiguous byte ranges, transfers those ranges as large aggregated blocks into GPU memory, and then instantiates framework tensors on top of the raw device buffers through DLPack. Because the I/O layer no longer cares about individual tensor boundaries, it can use many parallel reads, NUMA-aware placement, and peer-to-peer DMA where available, with a POSIX pread plus device-copy fallback elsewhere. Tensor-parallel sharding is moved onto the GPUs: files are read onto a round-robin set of devices and then redistributed with collective broadcast and scatter operations. On the tested models the loader outperforms safetensors 0.4.3 by 4.8x to 7.5x, approaches the measured storage ceiling (26.4 GB/s for Llama-70B on NVMe SSDs whose maximum is 28 GB/s), reduces host CPU and page-cache use, and cuts startup time in an existing open-source inference server by about 2 to 2.6 times.

Load-bearing premise

The load-bearing premise is that tensors created by wrapping raw GPU buffers via DLPack, after the alignment-fix copies, are functionally identical to tensors produced by the original safetensors deserializer; the paper's reported experiments run inference but do not compare the loaded tensors numerically against the baseline.

Editorial extensions

If this is right

  • Model startup in inference servers drops from minute-scale to second-scale: the paper reports 4.8x to 7.5x faster loading on Llama, Falcon, and Bloom checkpoints.
  • Storage is the bottleneck the loader actually exercises: measured NVMe throughput reaches 26.4 GB/s for Llama-70B, close to the 28 GB/s device ceiling, where the baseline used at most 5 GB/s.
  • Host resources are freed during startup: page-cache footprint and kernel CPU usage are sharply reduced, and with peer-to-peer DMA the host bounce buffer is eliminated.
  • Integration is cheap: replacing the safetensors weight-loading code in an existing open-source inference server required 42 lines of Python and cut startup time by about 2 to 2.6 times on tested Llama-2 configurations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the equivalence of DLPack-wrapped buffers holds generally, the same aggregate-then-wrap strategy should transfer to any serialized tensor format whose payloads are contiguous with known offsets, not just safetensors.
  • The reported speedups are tied to high-bandwidth NVMe storage and GPU interconnects; on slower storage or DRAM-backed tmpfs the PCIe bus becomes the limit, so the headline 4.8x to 7.5x figures should not be expected on every deployment.
  • The alignment-fix bounce pass is a workaround for a format-level gap; writing safetensors files with padded, aligned headers at serialization time would remove those extra GPU copies and make the peer-to-peer DMA path faster and simpler.
  • A direct element-wise comparison of loaded weights between the original loader and fastsafetensors would settle the correctness question more strongly than the single inference run reported, and would be a cheap integration test for adopters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper describes fastsafetensors, a Python library that changes the safetensors loading path from per-tensor deserialization in host memory to aggregated file-to-GPU copying followed by direct tensor instantiation via DLPack. It also offloads sharding and type conversion to GPU collectives and optionally uses GPUDirect Storage. The authors report 4.8x-7.5x loading time reductions for Llama-7B/13B/70B, Falcon-40B, and Bloom-176B compared with safetensors 0.4.3, and they evaluate resource utilization, GDS trade-offs under NUMA topologies and co-located jobs, and a vLLM integration.

Significance. If the speedups and correctness are confirmed, this is a practically useful systems contribution: it attacks a real pain point in LLM inference server startup, uses a format-compatible approach with open-source code, and provides useful engineering data on GDS. The paper's baseline is the current safetensors release and the measurements are direct performance comparisons, so the central speedup claim is not circular. The main missing piece is a correctness check that the loaded tensors are identical to those produced by the baseline, and the quantitative claim would be strengthened by repeated trials.

major comments (2)
  1. [Section III-A, III-B, IV-A] The central speedup claim is only meaningful if fastsafetensors produces tensors that are functionally identical to those produced by safetensors 0.4.3. Section III-A instantiates tensors by DLPack-wrapping raw CUDA buffers and computing strides from shape and dtype under a PyTorch-layout assumption, while Section III-B copies tensors through a GPU bounce buffer to fix misaligned offsets caused by odd-sized headers. An error in offset calculation, stride computation, or realignment would silently corrupt model weights while leaving load times unchanged. The only correctness check reported in Section IV-A is that a single TGIS prompt does not trigger an out-of-memory error; the paper never compares loaded tensors or inference outputs against the baseline loader. I request a numerical equivalence test: load the same checkpoint with both libraries, assert bitwise equality of every tensor (including post-sharding tensors), and compare generated tokens or logits for a fixed prompt.
  2. [Section IV-D, Figure 10, Table II] The headline 4.8x-7.5x speedup range is presented without repeated trials, error bars, or statistical tests. Model loading on shared systems with NVMe, page cache, and GPU state can vary run to run, so a single measurement per configuration does not establish the precision of the claimed range. Please report at least three to five runs per configuration with mean and standard deviation (or min/max), and state the number of trials explicitly for both the standalone loader and the vLLM startup times in Table II.
minor comments (4)
  1. [Throughout] The manuscript contains typos and inconsistent terminology, including 'usagses', 'cahce', 'experiemnts', 'concrreunt', 'toplogy', 'refrence', 'hus', and inconsistent 'Pytorch' versus 'PyTorch' spellings; these should be corrected.
  2. [Section III-C, Figures 8-9] The API name is inconsistent: the text says copy_file_to_device() while the code examples call copy_files_to_device(); please align the method name between the prose and the listings.
  3. [Figure 10] Figure 10a mixes elapsed-time bars and speedup bars without clearly stating the denominator; the caption should specify which configuration each speedup is normalized against and whether the 'gds' and 'nogds' labels apply to fastsafetensors only.
  4. [Section IV-F, Figure 15a] The tmpfs comparison changes both the storage medium and the GDS setting at once (tmpfs without GDS versus NVMe with GDS), which makes the claim that PCIe bandwidth is the limiting factor harder to assess; a controlled comparison would clarify the conclusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the reported speedups are direct wall-clock benchmarks against the external safetensors 0.4.3 baseline, not fitted or self-referential results.

full rationale

The paper's central claim is an empirical performance comparison, not a derived or fitted result. fastsafetensors is a new deserializer, and the 4.8x to 7.5x speedups (Abstract; Section IV-D, Figure 10a) are measured loading times for Llama, Falcon, and Bloom against safetensors 0.4.3, which is an external baseline library. No parameter is fitted to a subset of the data and then relabeled as a prediction; no equation in the paper reduces to its own output; and no uniqueness theorem is invoked to force a design choice. The minor self-citations (Column Cache [36] and Granite Code Models [3]) appear only as related-work context and application examples and are not load-bearing for the performance claim. The main validity concern is correctness, not circularity: Section IV-A checks only that a TGIS prompt does not trigger OOM, and Section III-B's bounce-buffer realignment for odd-sized headers is not validated by numerical tensor comparison against the baseline. That is an unvalidated precondition, but it does not make the performance measurement circular.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper is an empirical systems study; its central speedup claim rests on configuration choices and hardware/software assumptions rather than on fitted mathematical derivations.

free parameters (4)
  • I/O thread count cap = 80% of physical CPUs in a NUMA node
    Chosen to keep transfer block sizes large for GDS; tuned by hand, not fitted to data.
  • Bounce buffer size per thread = 160 MB
    Used for fallback DMA when GDS disabled; arbitrary size, affects performance.
  • GPU deserialization buffer size per file = 10 GB
    Buffer allocated per GPU for direct deserialization in experiments; a configuration choice.
  • Transfer block partitioning strategy = not specified
    The paper does not detail how files are partitioned into transfer blocks; this affects I/O efficiency.
assumptions (5)
  • domain assumption Safetensors files have a JSON header followed by the concatenated raw bytes of all tensors, with per-tensor offsets known in advance.
    The whole design depends on the format spec as documented in Section II-A.
  • domain assumption DLPack can create tensor objects in PyTorch by wrapping a raw CUDA buffer with the given shape, dtype, and strides.
    The library uses DLPack to instantiate tensors on GPU memory (Section III-A); if this fails for some dtypes or layouts, loading breaks.
  • domain assumption GPU memory is large enough to hold the model weights plus temporary buffers used for shuffling and alignment fixes.
    The design transfers whole files to GPU memory and allocates extra buffers for shuffling (Section IV-E); OOM would break loading.
  • domain assumption GPUDirect Storage and the underlying cuFile APIs behave as documented by NVIDIA when enabled.
    GDS performance and correctness are taken as given (Section III-A); the paper notes setup requirements and fallbacks.
  • domain assumption The benchmark workload (standalone loader and TGIS startup with a single prompt) represents realistic inference server startup behavior.
    The speedups are measured on this workload (Section IV-A/B); if real workloads differ, the reported gains may not generalize.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Speeding up Model Loading with fastsafetensors." pith.science (2026). https://pith.science/paper/CV7QSPL5

@misc{pith2026250523072,
  author       = {Pith},
  title        = {Pith review of: Speeding up Model Loading with fastsafetensors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CV7QSPL5}},
  note         = {Machine review of arXiv:2505.23072}
}
read the original abstract

The rapid increases in model parameter sizes introduces new challenges in pre-trained model loading. Currently, machine learning code often deserializes each parameter as a tensor object in host memory before copying it to device memory. We found that this approach underutilized storage throughput and significantly slowed down loading large models with a widely-used model file formats, safetensors. In this work, we present fastsafetensors, a Python library designed to optimize the deserialization of tensors in safetensors files. Our approach first copies groups of on-disk parameters to device memory, where they are directly instantiated as tensor objects. This design enables further optimization in low-level I/O and high-level tensor preprocessing, including parallelized copying, peer-to-peer DMA, and GPU offloading. Experimental results show performance improvements of 4.8x to 7.5x in loading models such as Llama (7, 13, and 70 billion parameters), Falcon (40 billion parameters), and the Bloom (176 billion parameters).

Figures

Figures reproduced from arXiv: 2505.23072 by the authors.

Figure 1
Figure 1. Overview of the safetensors format: The file is divided [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 4
Figure 4. Tensor copy flow. Header Rank 0 GPU 0 GPU 1 GPU 2 GPU 3 Tensor A [,0:1] Tensor A [,1:2] Tensor A [,2:3] Tensor A [,3:4] Rank 1 Rank 2 Rank 3 Tensor A, shape: (4, 4) Tensor A [,0:1] Tensor A [,1:2] Tensor A [,2:3] Tensor A [,3:4] File A [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Tensor sharding. the file are mapped to host memory, instantiated, and then copied to GPU memory. Existing inference servers are often implemented in Python, where tensor instantiation is processed sequentially. Furthermore, due to the on-demand loading of files via mmap, file prefetching relies on generic heuristics in Linux, which makes it challenging to achieve optimal performance with high-performance storage su… view at source ↗
Figures from the paper (9 more)
Figure 3
Figure 3. Figure 3: Resource utilization of TGIS. model loading as shown in Figure 3e and 3g. We observed GPU and NVLink utilization only after starting inference for all the models. The GPU memory usages incrementally increased at the same pace as page cahce usages, as illustrated in Fig…
Figure 6
Figure 6. Figure 6: Batching of file I/O and tensor instantiation with [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 8
Figure 8. Figure 8: Example of single GPU application. It reads [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Example of multi-GPU application. It reads files to [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 10
Figure 10. Figure 10: Performance of fastsafetensors. processing and sharding policy, i.e., partitioned dimension. As described in the previous section, fastsafetensors first copies files to GPU memory and then relocates partitioned tensors with shuffling. In contrast, the default safetens…
Figure 11
Figure 11. Figure 11: Host CPU usages of fastsafetensors. Figure 10a shows the elapsed time of our standalone loader with each configuration and the speedups of fastsafetensors compared to the default safetensors library with GDS enabled. Fastsafetensors improved both single- and multi-GPU…
Figure 13
Figure 13. Figure 13: NVMe SSD read throughput of fastsafetensors. [PITH_FULL_IMAGE:figures/full_fig_p008_13.png]
Figure 15
Figure 15. Figure 15: Performance speedup under different situations: Each [PITH_FULL_IMAGE:figures/full_fig_p009_15.png]
Figure 14
Figure 14. Figure 14: GPU resource usages of fastsafetensors. usage for iterative decoder processing to generate 128 tokens with 263 input tokens. Shuffling requires additional memory buffering at GPUs, but it was not notably high since it kept reusing GPU memory with the Torch memory allo…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Serialized Bridge: Understanding and Recovering LLM Serving Performance under Blackwell GPU Confidential Computing

    cs.DC 2026-06 unverdicted novelty 7.0 of 10

    Under GPU-CC, LLM serving losses come from a serialized VM–GPU bridge, not compute; simple scheduling and loader changes recover most of the gap on Blackwell.

  2. InstantInfer: Enabling Fast LLM Cold Start with Communicating Finite Automata

    cs.DC 2026-07 conditional novelty 5.0 of 10

    InstantInfer refactors vLLM's cold start into a concurrent state-machine pipeline, speeding up startup by up to 7.2×.

Reference graph

Works this paper leans on

47 extracted references · 30 canonical work pages · cited by 2 Pith papers

  1. [1]

    Introducing ChatGPT,

    OpenAI, “Introducing ChatGPT,” 2022. [Online]. Available: https: //openai.com/index/chatgpt/

    work page 2022

  2. [2]

    Introducing Gemini: our largest and most capable AI model,

    S. Pichai and D. Hassabis, “Introducing Gemini: our largest and most capable AI model,” 2023. [Online]. Available: https: //blog.google/technology/ai/google-gemini-ai/

    work page 2023

  3. [3]

    Granite Code Models: A Family of Open Foundation Models for Code Intelligence,

    M. Mishra, M. Stallone, G. Zhang, Y . Shen, A. Prasad, A. M. Soria, M. Merler, P. Selvam, S. Surendran, S. Singh, M. Sethi, X.-H. Dang, P. Li, K.-L. Wu, S. Zawad, A. Coleman, M. White, M. Lewis, R. Pavuluri, Y . Koyfman, B. Lublinsky, M. de Bayser, I. Abdelaziz, K. Basu, M. Agarwal, Y . Zhou, C. Johnson, A. Goyal, H. Patel, Y . Shah, P. Zerfos, H. Ludwig,...

    work page

  4. [4]

    The Shift from Models to Compound AI Systems,

    M. Zaharia, O. Khattab, L. Chen, J. Q. Davis, H. Miller, C. Potts, J. Zou, M. Carbin, J. Frankle, N. Rao, and A. Ghodsi, “The Shift from Models to Compound AI Systems,” 2024. [Online]. Available: https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/

    work page 2024

  5. [5]

    FlashAttention: Fast and memory-efficient exact attention with IO-awareness,

    T. Dao, D. Y . Fu, S. Ermon, A. Rudra, and C. Ré, “FlashAttention: Fast and memory-efficient exact attention with IO-awareness,” in Advances in Neural Information Processing Systems (NeurIPS ’22) , 2022

    work page 2022

  6. [6]

    FlashAttention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “FlashAttention-2: Faster attention with better parallelism and work partitioning,” in International Conference on Learning Represen- tations (ICLR ’24) , 2024

    work page 2024

  7. [7]

    Efficient Memory Management for Large Language Model Serving with PagedAttention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient Memory Management for Large Language Model Serving with PagedAttention,” in Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23) , 2023, p. 611–626

    work page 2023

  8. [8]

    Orca: A Distributed Serving System for Transformer-Based Generative Models,

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A Distributed Serving System for Transformer-Based Generative Models,” in 16th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI ’22) , 2022, pp. 521–538

    work page 2022

Show all 47 references
  1. [9]

    Accelerating Production LLMs with Combined Token/Embedding Speculators,

    D. Wertheimer, J. Rosenkranz, T. Parnell, S. Suneja, P. Ranganathan, R. Ganti, and M. Srivatsa, “Accelerating Production LLMs with Combined Token/Embedding Speculators,” 2024. [Online]. Available: https://arxiv.org/abs/2404.19124

    2024 arXiv

  2. [10]

    DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving,

    Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving,” 2024. [Online]. Available: https://arxiv.org/abs/2401.09670

    2024 arXiv

  3. [11]

    Taming throughput-latency tradeoff in llm inference with sarathi-serve,

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, and R. Ramjee, “Taming throughput-latency tradeoff in llm inference with sarathi-serve,” Proceedings of 18th USENIX Symposium on Operating Systems Design and Implementation, (OSDI ’24) , 2024

    2024

  4. [12]

    Decrease PyTorch Model Load Times with CoreWeave’s Tensorizer,

    N. Pratt and R. Talari, “Decrease PyTorch Model Load Times with CoreWeave’s Tensorizer,” 2024. [Online]. Avail- able: https://www.coreweave.com/blog/coreweaves-tensorizer-decrease- pytorch-model-load-times

    2024

  5. [13]

    ServerlessLLM: Locality-Enhanced Serverless Inference for Large Language Models,

    Y . Fu, L. Xue, Y . Huang, A.-O. Brabete, D. Ustiugov, Y . Patel, and L. Mai, “ServerlessLLM: Locality-Enhanced Serverless Inference for Large Language Models,” 2024. [Online]. Available: https: //arxiv.org/abs/2401.14351

    2024 arXiv

  6. [14]

    Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM,

    D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V . A. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM,” 2021. [Online]. Availa...

    2021 arXiv

  7. [15]

    Safetensors,

    Hugging Face, “Safetensors,” 2024. [Online]. Available: https:// huggingface.co/docs/safetensors/

    2024

  8. [16]

    Models – Hugging Face,

    “Models – Hugging Face,” 2025. [Online]. Available: https:// huggingface.co/models?library=safetensors

    2025

  9. [17]

    pickle — Python object Serialization,

    Python Software Foundation, “pickle — Python object Serialization,”

  10. [18]

    Pytorch,

    The Linux Foundation, “Pytorch,” 2024. [Online]. Available: https: //pytorch.org/

    2024

  11. [19]

    Available: https://docs .python.org/3/library/pickle.html

    [Online]. Available: https://docs .python.org/3/library/pickle.html

  12. [20]

    ZeRO- Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning,

    S. Rajbhandari, O. Ruwase, J. Rasley, S. Smith, and Y . He, “ZeRO- Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning,” 2021. [Online]. Available: https://arxiv .org/abs/2104.07857

    2021 arXiv

  13. [21]

    TensorFlow: A System for Large-Scale Machine Learning,

    M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V . Vasudevan, P. Warden, M. Wicke, Y . Yu, and X. Zheng, “TensorFlow: A System for Large-Sca...

    2016

  14. [22]

    ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development,

    B. Wan, M. Han, Y . Sheng, Y . Peng, H. Lin, M. Zhang, Z. Lai, M. Yu, J. Zhang, Z. Song, X. Liu, and C. Wu, “ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development,”

  15. [23]

    PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel,

    Y . Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y . Hao, A. Mathews, and S. Li, “PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel,”

  16. [24]

    NVIDIA Magnum IO GPUDirect Storage Design Guide,

    NVIDIA, “NVIDIA Magnum IO GPUDirect Storage Design Guide,”

  17. [25]

    Llama 2: Open Foundation and Fine-Tuned Chat Models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V . Goswami, N. Goyal, A. Hartshorn, S. Ho...

    2023 arXiv

  18. [26]

    Available: https://arxiv .org/abs/2407.20143

    [Online]. Available: https://arxiv .org/abs/2407.20143

    arXiv

  19. [27]

    Welcome to DLPack’s documentation! — DLPack 0.6.0 documentation,

    DLPack contributors, “Welcome to DLPack’s documentation! — DLPack 0.6.0 documentation,” 2022. [Online]. Available: https: //dmlc.github.io/dlpack/latest/

    2022

  20. [28]

    huggingface/text-generation-inference: Large Language Model Text Generation Inference,

    “huggingface/text-generation-inference: Large Language Model Text Generation Inference,” 2024. [Online]. Available: https://github .com/ huggingface/text-generation-inference

    2024

  21. [29]

    vllm-project/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs,

    “vllm-project/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs,” 2024. [Online]. Available: https: //github.com/vllm-project/vllm

    2024

  22. [30]

    SGLang: Efficient Execution of Structured Language Model Programs,

    L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y . Sheng, “SGLang: Efficient Execution of Structured Language Model Programs,”

  23. [31]

    The Falcon Series of Open Language Models,

    E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, É. Goffinet, D. Hesslow, J. Launay, Q. Malartic, D. Mazzotta, B. Noune, B. Pannier, and G. Penedo, “The Falcon Series of Open Language Models,” 2023. [Online]. Available: https://arxiv.org/abs/2311.16867

    2023 arXiv

  24. [32]

    BLOOM: A 176B-Parameter Open-Access Multilingual Language Model,

    T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ili ´c, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V . del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Ma...

    2023 arXiv

  25. [33]

    PEP 703 — Making the Global Interpreter Lock Optional in CPython,

    Sam Gross, “PEP 703 — Making the Global Interpreter Lock Optional in CPython,” 2023, accessed: 2025-05-20. [Online]. Available: https://peps.python.org/pep-0703/

    2023

  26. [34]

    SPIN: Seam- less Operating System Integration of Peer-to-Peer DMA Between SSDs and GPUs,

    S. Bergman, T. Brokhman, T. Cohen, and M. Silberstein, “SPIN: Seam- less Operating System Integration of Peer-to-Peer DMA Between SSDs and GPUs,” in 2017 USENIX Annual Technical Conference (USENIX ATC ’17), 2017, pp. 167–179

    2017

  27. [35]

    How beneficial is peer-to-peer DMA?

    R. Nakamura, Y . Kuga, and K. Akashi, “How beneficial is peer-to-peer DMA?” in Proceedings of the 11th ACM SIGOPS Asia-Pacific Workshop on Systems (APSys ’20) , 2020, p. 25–32

    2020

  28. [36]

    Available: https://arxiv .org/abs/2312.07104

    [Online]. Available: https://arxiv .org/abs/2312.07104

    arXiv

  29. [37]

    Rapid Data Pre- Processing with NVIDIA DALI,

    J. A. Guirao, R. Banas, K. Ł˛ ecki, J. Lisiecki, A. Wolant, M. Zientkiewicz, K. Tokarski, and M. Szołucha, “Rapid Data Pre- Processing with NVIDIA DALI,” 2021. [Online]. Available: https:// developer.nvidia.com/blog/rapid-data-pre-processing-with-nvidia-dali/

    2021

  30. [38]

    Accelerate AI and ML workloads with OCI, NVIDIA Magnum IO GPUDirect Storage, and IBM Storage Scale,

    P. Valdria, “Accelerate AI and ML workloads with OCI, NVIDIA Magnum IO GPUDirect Storage, and IBM Storage Scale,” 2023. [Online]. Available: https://blogs .oracle.com/cloud-infrastructure/post/ accelerate-ai-ml-workloads-oci-nvidia-ibm

    2023

  31. [39]

    FP8 Formats for Deep Learning,

    P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, S. Ha, A. Heinecke, P. Judd, J. Kamalu, N. Mellempudi, S. Oberman, M. Shoeybi, M. Siu, and H. Wu, “FP8 Formats for Deep Learning,” 2022. [Online]. Available: https://arxiv.org/abs/2209.05433

    2022 arXiv

  32. [40]

    Efficient post-training quantization with fp8 formats,

    H. Shen, N. Mellempudi, X. He, Q. Gao, C. Wang, and M. Wang, “Efficient post-training quantization with fp8 formats,” 2024. [Online]. Available: https://arxiv.org/abs/2309.14592

    2024 arXiv

  33. [41]

    FP8 Quantization: The Power of the Exponent,

    A. Kuzmin, M. V . Baalen, Y . Ren, M. Nagel, J. Peters, and T. Blankevoort, “FP8 Quantization: The Power of the Exponent,” 2024. [Online]. Available: https://arxiv.org/abs/2208.09225

    2024 arXiv

  34. [42]

    Column Cache: Buffer Cache for Columnar Storage on HDFS,

    T. Yoshimura, T. Chiba, and H. Horii, “Column Cache: Buffer Cache for Columnar Storage on HDFS,” in 2018 IEEE International Conference on Big Data (Big Data ’18) , 2018, pp. 282–291

    2018

  35. [43]

    Teraheap: Reducing memory pressure in managed big data frameworks,

    I. G. Kolokasis, G. Evdorou, S. Akram, C. Kozanitis, A. Papagian- nis, F. S. Zakkak, P. Pratikakis, and A. Bilas, “Teraheap: Reducing memory pressure in managed big data frameworks,” in Proceedings of the 28th ACM International Conference on Architectural Support for Programmi...

    2023

  36. [44]

    Accelerating multilingual applications with in-memory array sharing,

    M. Nozawa, S. Imamura, and K. Kono, “Accelerating multilingual applications with in-memory array sharing,” in 2023 IEEE International Conference on Big Data (BigData ’23) , 2023, pp. 255–262

    2023

  37. [2023]

    Available: https://arxiv .org/abs/2304.11277

    [Online]. Available: https://arxiv .org/abs/2304.11277

    arXiv

  38. [2024]

    Available: https://arxiv .org/abs/2405.04324

    [Online]. Available: https://arxiv .org/abs/2405.04324

    arXiv

  39. [2025]

    Available: https://docs .nvidia.com/gpudirect-storage/ design-guide/index.html

    [Online]. Available: https://docs .nvidia.com/gpudirect-storage/ design-guide/index.html

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.