pith. sign in

arxiv: 2110.15032 · v6 · pith:6UEZBTKFnew · submitted 2021-10-28 · 💻 cs.DC · cs.AI· cs.LG

OneFlow: Redesign the Distributed Deep Learning Framework from Scratch

Jinhui Yuan , Xinqi Li , Cheng Cheng , Juncheng Liu , Ran Guo , Shenghang Cai , Chi Yao , Fei Yang
show 4 more authors
Xiaodong Yi Chuan Wu Haoran Zhang Jie Zhao
This is my paper
classification 💻 cs.DC cs.AIcs.LG
keywords parallelismdeepdistributedoneflowframeworkslearningmodeldata
0
0 comments X
read the original abstract

Deep learning frameworks such as TensorFlow and PyTorch provide a productive interface for expressing and training a deep neural network (DNN) model on a single device or using data parallelism. Still, they may not be flexible or efficient enough in training emerging large models on distributed devices, which require more sophisticated parallelism beyond data parallelism. Plugins or wrappers have been developed to strengthen these frameworks for model or pipeline parallelism, but they complicate the usage and implementation of distributed deep learning. Aiming at a simple, neat redesign of distributed deep learning frameworks for various parallelism paradigms, we present OneFlow, a novel distributed training framework based on an SBP (split, broadcast and partial-value) abstraction and the actor model. SBP enables much easier programming of data parallelism and model parallelism than existing frameworks, and the actor model provides a succinct runtime mechanism to manage the complex dependencies imposed by resource constraints, data movement and computation in distributed deep learning. We demonstrate the general applicability and efficiency of OneFlow for training various large DNN models with case studies and extensive experiments. The results show that OneFlow outperforms many well-known customized libraries built on top of the state-of-the-art frameworks. The code of OneFlow is available at: https://github.com/Oneflow-Inc/oneflow.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MAGI-1: Autoregressive Video Generation at Scale

    cs.CV 2025-05 unverdicted novelty 6.0

    MAGI-1 is a 24B-parameter autoregressive video world model that predicts denoised frame chunks sequentially with increasing noise to enable causal, scalable, streaming generation up to 4M token contexts.

  2. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

    cs.DC 2023-04 unverdicted novelty 6.0

    PyTorch Fully Sharded Data Parallel enables training of significantly larger models than Distributed Data Parallel with comparable speed and near-linear TFLOPS scaling.

  3. DPIFrame: A Dual-Level Parallelism Acceleration Framework for CTR Model Inference

    cs.DC 2026-06 unverdicted novelty 5.0

    DPIFrame introduces intra- and inter-module parallel architecture, anticipatory multi-table embedding lookup, and breadth-first GPU stream scheduling, reporting 23x embedding latency reduction and up to 5.83x overall ...

  4. AAFLOW: Scalable Patterns for Agentic AI Workflows

    cs.DC 2026-05 unverdicted novelty 5.0

    AAFLOW is a unified distributed runtime that models agentic workflows as operators with a zero-copy data plane using Apache Arrow and Cylon, achieving up to 4.64x pipeline speedup through improved data flow and batching.

  5. A Survey of Large Language Models

    cs.CL 2023-03 accept novelty 3.0

    This survey reviews the background, key techniques, and evaluation methods for large language models, emphasizing emergent abilities that appear at large scales.