AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training

Ansheng You; Chenyi Pan; Chunsheng Li; Chunshi Deng; Guang Yang; Haibo Wang; Huazhong Ji; Jianping Wu; Jing Wang; Kui Luo

REVIEW 5 major objections 6 minor 4 cited by

AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A streaming asynchronous RL framework for LLM post-training reports an average 1.59x throughput gain over the state of the art.

desk verdict Useful systems engineering for async RL post-training, but the headline 1.59x against an author-adapted verl on Ascend is not auditable without code, patch, and longer runs. read the letter →

arxiv 2507.01663 v1 pith:F5FWAOGQ submitted 2025-07-02 cs.LG cs.AI

Zhenyu Han , Ansheng You , Haibo Wang , Kui Luo , Guang Yang , Wenqi Shi , Menglong Chen , Sicheng Zhang

show 11 more authors

Zeshun Lan Chunshi Deng Huazhong Ji Wenjie Liu Yu Huang Yixiang Zhang Chenyi Pan Jing Wang Xin Huang Chunsheng Li Jianping Wu

This is my paper · ORCID

classification cs.LGcs.AI

keywords asynchronousreinforcementlearningLLMpost-trainingGRPOdistributeddataflowschedulingstreamingdataloadertask-separatedRLdelayedparameterupdatethroughputscaling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a task-separated, fully streamed reinforcement-learning framework can beat the dominant task-collocated design for LLM post-training at scale. It introduces a distributed streaming dataloader that routes data between inference and training engines on demand, plus a producer-consumer asynchronous workflow that lets rollout workers keep generating with slightly stale weights. On clusters from 32 to 1024 NPUs, the paper measures an average 1.59x throughput improvement over a state-of-the-art baseline, with the largest gain of 2.03x at 256 NPUs, while reward and response-length trajectories stay close to the synchronous workflow. A sympathetic reader would care because post-training is becoming the compute bottleneck for reasoning models, and the result suggests the bottleneck is schedulable rather than fundamental.

What carries the argument

The load-bearing object is TransferQueue, a distributed streaming dataloader with a decoupled control plane and data plane. Each RL task gets a controller that tracks which samples and columns are ready and unconsumed; storage units hold rows of a 2D columnar table addressed by global indices; consumers request micro-batches and read only the columns they need. This removes explicit cross-task dependency definitions and enables dynamic load balancing and pipeline overlap. The second mechanism is a producer-consumer asynchronous workflow with delayed parameter update: rollout workers continue generating on old weights while new weights arrive asynchronously, and the weight swap happens only when the current generation iteration ends, keeping the pipeline within a one-step staleness bound. A hybrid analytical-plus-profiling cost model searches the resource allocation that minimizes projected end-to-end time.

What would settle it

Run AsyncFlow and the baseline on identical NPU clusters for a full GRPO run with the baseline's native configuration and native sequence lengths, measuring end-to-end tokens per second and final reward; a throughput premium below 1.59x or a reward/response-length divergence beyond the reported variance would refute the central claim.

Watch

Extended reading notes

Core claim

The central claim is that the end-to-end RL post-training workflow can be restructured as a stream: instead of defining fixed data-dependency chains between tasks and waiting for whole batches, AsyncFlow moves fine-grained samples through a distributed queue whose per-task controllers hold metadata and dynamically assemble micro-batches for whichever consumer asks. On top of this, the paper argues that one-step asynchronization between actor rollout and actor update—rollout continues on old weights while new weights are transmitted and swapped at the next iteration boundary—removes warm-up and cool-down bubbles without measurably hurting convergence. The measured consequence is a consistent throughput advantage over the task-collocated state of the art across every cluster size tested, growing with scale and reaching up to 2.03x, with comparable reward and response length under equal clock time.

Load-bearing premise

The speedup claim assumes the ported baseline was run at its intended efficiency on the same hardware, and that a 10-20 iteration window captures steady-state throughput; if the baseline port is handicapped or the window is unrepresentative, the 1.59x average is not established.

Editorial extensions

If this is right

Task-separated RL frameworks can sustain high utilization at scale: the paper reports scaling linearities of 0.65 and 0.88 when the cluster expands 16x.
One-step asynchronization between rollout and update is enough to eliminate most pipeline bubbles without a reward penalty in the tested regime.
Streaming dataflow with centralized metadata lets faster workers pull more samples, so heterogeneous load and variable response lengths are absorbed automatically.
Because the core is exposed through service-oriented adapters, existing training and inference engines can be plugged in without rewriting the RL algorithm.
The measured gain grows with cluster size, suggesting the architecture's advantage is largest exactly in the industrial-scale regime where post-training is most expensive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's 10-20 iteration measurement window, an untested implication is that the 1.59x average could shrink or grow as sequence lengths lengthen and weight-sync frequency changes in later training; a steady-state run over hundreds of iterations would settle this.
The same TransferQueue scheduling could plausibly generalize beyond GRPO to PPO and multi-reward setups, but only if per-task controllers remain cheap enough when the task count grows; the paper notes PPO support is still under development.
The sub-step asynchrony sketched as future work would push staleness below one training step; whether convergence remains stable at that tighter threshold is an open empirical question the paper does not answer.
If the hybrid cost model's profiling component is accurate, the same resource-planning search could be reused to choose allocations on heterogeneous hardware, where dynamic load balancing should matter even more; the paper does not test this.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Take a look at this if you work on RL post-training systems. The nice part is the architecture: TransferQueue, with its control/data plane split, 2D columnar data layout, and per-sample consumption metadata, is a clean way to stream experience through distributed RL tasks. The delayed parameter update trick, deferring weight load to host until current generation ends, is a sensible one-step asynchronization that they honestly note is conceptually similar to StreamRL. The ablation is internally consistent: TransferQueue gives 2.01x over a sequential task-separated baseline, and async opt adds 36.3%. So the components do something.

The soft spot is the headline number. The 1.59x average throughput over verl hangs on their own adaptation of verl to Ascend at commit d13434f, and they give no patch, no config, no error bars, and only 10-20 iterations per run. The stress-test note is right: if the porting path leaves verl without its hybrid engine overlap, the comparison could be against a handicapped baseline. Also no absolute throughput numbers, so the reader can't calibrate. The convergence evidence is qualitative and one 7B/16-NPU run, but that's secondary; the throughput baseline is the load-bearing claim. They also defer the sub-step asynchrony to future work, which is fine if it's not sold as a contribution, but the text oversells it slightly. No code release, which makes the central measurement uncheckable.

That said, the reader's take is fair; this is a conditional accept rather than a reject. The paper is a serious systems contribution with clear diagrams, honest related-work discussion, and an internally consistent ablation. The evaluation needs more rigor before the 1.59x is taken at face value.

I'd send it to review, with the expectation that reviewers ask for the verl patch, longer steady-state runs, error bars, and ideally a comparison against StreamRL or at least a discussion of why it's infeasible on Ascend. The paper will be useful to anyone building task-separated RL infrastructure, even if only as a source of design patterns. For my own work, I'd cite it for the TransferQueue design and the ablation, not for the baseline comparison.

Referee Report

5 major / 6 minor

Summary. The paper proposes AsyncFlow, an asynchronous streaming reinforcement learning (RL) framework for LLM post-training, aimed at overcoming scalability and resource-idling problems of task-collocated and task-separated frameworks. The core technical contributions are: (i) TransferQueue, a distributed streaming dataloader with decoupled control and data planes that provides centralized data management, dynamic load balancing, and fine-grained pipeline overlap; (ii) a producer–consumer-based asynchronous workflow with a delayed parameter update mechanism that reduces warm-up and cool-down bubbles while bounding staleness; and (iii) a service-oriented interface layer that decouples the framework from specific training/inference engines. The authors report an average 1.59x throughput improvement over the verl baseline across 32–1024 Ascend NPUs (peak 2.03x at 256 NPUs for the 7B model), along with a four-point ablation showing 2.01x speedup from TransferQueue and 2.74x with the full asynchronous workflow. Convergence is examined via reward and response-length curves on a 7B/16-NPU setup.

Significance. If substantiated, the result is important for industrial-scale LLM post-training: task-separated architectures that achieve high utilization through streaming and controlled asynchrony could remove a major practical bottleneck. The paper's architectural ideas—especially TransferQueue's separation of control and data planes, dynamic fine-grained scheduling, and delayed parameter updates—are well motivated and align with recent trends in RL systems (e.g., StreamRL, AReaL). The ablation study is internally consistent and provides a useful decomposition of the gains. However, the headline 1.59x speedup depends on a baseline comparison that is not fully auditable, and the measurement methodology lacks error bars and absolute throughput values. The convergence analysis is also thin. These issues are fixable within the manuscript's scope, but they currently weaken the central quantitative claim.

major comments (5)

[§6.1, §6.2] Baseline fairness for the headline 1.59x average throughput (and 2.03x peak) is not established. The verl baseline is described only as 'adapted to Ascend NPUs' at commit d13434f, with no patch, configuration, or deployment details. If the adaptation prevents verl from using its 3D-HybridEngine overlap or if the vLLM-Ascend integration is undertuned, the measured speedup may reflect a handicapped baseline rather than AsyncFlow's architectural advantage. Please provide the exact adaptation patch (or at least a detailed configuration description), the verl setting used, and, ideally, a comparison on a platform where verl runs natively.
[§6.2, Fig. 10] The throughput measurements are reported as averages over only 10–20 iterations, with no error bars, confidence intervals, or absolute throughput numbers. This makes it impossible to assess run-to-run variance or whether the short window captures steady-state behavior (e.g., after weight-sync intervals and with longer responses, which dominate later training). The scaling-efficiency 'linearity' values (0.65 and 0.88) are also undefined. Please report per-configuration absolute throughput (e.g., samples/s or tokens/s), variance or confidence intervals, and define the linearity metric explicitly.
[§6.5, Fig. 12] The convergence evidence is qualitative and limited to a single 7B/16-NPU run. The claim of 'negligible differences' in reward and a 'convergence trend' in response length is not supported by quantitative results—no final performance numbers, standard deviations, or multiple seeds are given. Since the asynchronous off-policy design is a core contribution with potential convergence implications, stronger evidence is needed: e.g., final reward and response-length statistics, held-out benchmark accuracy, and at least a second configuration (different model size or cluster scale).
[§6.3, Table 1] The internal ablation (baseline, +TransferQueue, +Async.Opt → 1.0, 2.01, 2.74) is plausible, but it compares AsyncFlow against a sequential task-separated baseline of the authors' own construction, not against the adapted verl. The relationship between that sequential baseline and verl is never established, so the decomposition of the 1.59x speedup into TransferQueue and async-optimization components cannot be audited. Please report the equivalent throughput of the adapted verl under the same measurement setup, or provide an explicit mapping between the ablation baseline and verl.
[§4.2.2, §1] The 'sub-step asynchronous workflow' (Fig. 8d) is described in the introduction and system sections as part of the framework's contribution, but §4.2.2 explicitly states: 'We leave the implementation details of this mechanism for an important future work.' Since the mechanism is neither implemented nor evaluated, claims about it should either be removed from the contribution summary or clearly framed as a design idea deferred to future work.

minor comments (6)

[Abstract, §6.2] The abstract uses 'AsynFlow' (one 'c') while the paper title and body use 'AsyncFlow' (two 'c's); please standardize the spelling.
[Fig. 10, Fig. 12] The figure captions and plot text for these figures contain garbled/unreadable characters (e.g., '/uni00000016/...'), making the reported data impossible to read. These figures must be regenerated with readable text.
[§6.1] The verl commit reference (d13434f, dated April 7, 2025) should include a URL or explicit repository location so that reviewers and readers can inspect the exact baseline version.
[§6.2] The term 'scaling efficiency' / 'linearity' is used without a formula or definition; please provide the exact computation (e.g., ratio of per-resource throughput at scale).
[§7.1] Reference [27] is cited as 'verl' in the text but listed as 'Hybridflow: A flexible and efficient rlhf framework'; please clarify the relationship between HybridFlow and verl and cite appropriately.
[§6.1] The statement 'Support for PPO is currently under development' is a limitation that should be acknowledged in the conclusion or future-work section, not only in the experimental setup.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central 1.59x throughput claim is an empirical measurement against an external baseline, not a derivation from its own inputs.

full rationale

AsyncFlow's central claim is an empirical systems measurement: AsyncFlow is compared to verl, an external state-of-the-art baseline, and the 1.59x figure is reported as measured throughput in Section 6.2. No fitted parameter is renamed as a prediction, and no self-derived normalization is used to construct the throughput numbers. The internal ablation baseline in Section 6.3 is a controlled variant with all proposed optimizations disabled; it is used to decompose the speedup, not to define the headline comparison. The one-step asynchronization convergence assumption is justified by external references [13, 38], not by a self-citation chain. The mention of MindSpeed-RL [6] and the statement that AsyncFlow will be integrated into it describe a deployment plan and do not carry the load of the throughput or convergence claims. Baseline fairness concerns, such as the undescribed Ascend adaptation of verl at commit d13434f, short 10-20 iteration measurement windows, and absent error bars, are correctness and auditability risks rather than circularity, because they do not make the result equal to its inputs by construction. No circular step can therefore be exhibited.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The framework introduces software components (TransferQueue, controllers) but no new physical or mathematical entities. The only load-bearing assumptions are the convergence-friendliness of one-step staleness, the accuracy of the resource simulator, and the representativeness of the short throughput measurement window.

assumptions (3)

domain assumption One-step asynchronization between actor rollout and actor update does not significantly degrade convergence.
Used to justify the delayed parameter update mechanism (Sec. 4.2.1 and 4.2.2, citing [13,38]); the paper does not prove this for GRPO/Qwen2.5 and only tests on 16 NPUs.
domain assumption The execution time simulator in Sec. 4.3 estimates task durations accurately enough to pick optimal resource allocation.
No validation of the simulator's accuracy is reported in the evaluation.
ad hoc to paper Throughput measurements over 10-20 iterations are representative of steady-state RL post-training.
The paper states it averages 10-20 iterations to reduce fluctuations but provides no analysis of steady-state behavior.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training." pith.science (2026). https://pith.science/paper/F5FWAOGQ

@misc{pith2026250701663,
  author       = {Pith},
  title        = {Pith review of: AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F5FWAOGQ}},
  note         = {Machine review of arXiv:2507.01663}
}

read the original abstract

Reinforcement learning (RL) has become a pivotal technology in the post-training phase of large language models (LLMs). Traditional task-colocated RL frameworks suffer from significant scalability bottlenecks, while task-separated RL frameworks face challenges in complex dataflows and the corresponding resource idling and workload imbalance. Moreover, most existing frameworks are tightly coupled with LLM training or inference engines, making it difficult to support custom-designed engines. To address these challenges, we propose AsyncFlow, an asynchronous streaming RL framework for efficient post-training. Specifically, we introduce a distributed data storage and transfer module that provides a unified data management and fine-grained scheduling capability in a fully streamed manner. This architecture inherently facilitates automated pipeline overlapping among RL tasks and dynamic load balancing. Moreover, we propose a producer-consumer-based asynchronous workflow engineered to minimize computational idleness by strategically deferring parameter update process within staleness thresholds. Finally, the core capability of AsynFlow is architecturally decoupled from underlying training and inference engines and encapsulated by service-oriented user interfaces, offering a modular and customizable user experience. Extensive experiments demonstrate an average of 1.59 throughput improvement compared with state-of-the-art baseline. The presented architecture in this work provides actionable insights for next-generation RL training system designs.

Figures

Figures reproduced from arXiv: 2507.01663 by the authors.

**Figure 1.** System overview of AsyncFlow framework. fully utilize all the computational resources due to scaling inefficiencies. While for task-separated frameworks, distinct RL tasks are assigned to different resource pools proportionally sized to their workload demands. In typical task-separated frameworks (e.g., OpenRLHF [5]), distributed resource management and execution tools (e.g., Ray [11]) are introduced to schedule R… view at source ↗

**Figure 3.** Architecture design of TransferQueue. Each DP group interacts with its corresponding TransferQueue controller to get the metadata upon request, then execute the read/write operation with storage units in the data plane. The dashed line represents metadata communication, while the solid line depicts the communication process with real data. All these interaction processes are encapsulated as a distributed streaming … view at source ↗

**Figure 4.** Data structure of TransferQueue. Each row represents a complete data sample, while each column represents the task-specific data component. Through the global index, we can accurately address the data samples across different storage units for concurrent read/write operations. 0 0->1 0->1 0 0 1 2 3 0 0 4 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0->1 1 1 0 1 2 3 0->1 0 4 5 0 Data Storage Unit #0 Data Storage Unit #1 Co… view at source ↗

Figures from the paper (8 more)

**Figure 5.** Figure 5: Metadata notification process of TransferQueue. Data storage units will broadcast the metadata (including the global index and corresponding data columns) to all the controllers. broadcast the corresponding row indices and column identifiers to all the controllers (re…

**Figure 6.** Figure 6: Scheduling process of TransferQueue controller. Red indexes demote that the corresponding data samples satisfies the RL task requirement, and the checkmark means the corresponding data has been consumed in earlier requests. We initialize distinct TransferQueue control…

**Figure 7.** Figure 7: Streaming pipeline overlapping. varying tasks, eliminating the need to manually reschedule the data streams. 4.2 Asynchronous Off-Policy Bubble Reduction 4.2.1 Basic asynchronous RL algorithm. Traditional RL algorithms for LLM post-training predominantly adopt a synchr…

**Figure 8.** Figure 8: Illustration of the asynchronous off-policy RL workflow. Leveraging the proposed delay parameter update mechanism in (c), AsyncFlow can extend the steady phase of the RL pipeline to further reduce pipeline bubbles. By sequentially enabling parameter updates for rollout…

**Figure 9.** Figure 9: Architecture of service-oriented user interface. method provides block-level performance by actually running training and inference tasks. It provides an accurate evaluation with the cost of a much higher time consumption. Leveraging the hybrid cost model, we ensure a…

**Figure 10.** Figure 10: End-to-end throughput and scalability analysis across varying cluster and model sizes. We report the average throughput across 10-20 iterations to reduce measurement fluctuations at the beginning. This layer ensures seamless integration of diverse training/inference …

**Figure 11.** Figure 11: AsyncFlow workflow of each training and inference instances. We showcase the 32B model with 512 NPUs, with iteration 0-3. 0 200 400 600 800 1000 0.0 0.2 0.4 0.6 Reward 0 200 400 600 800 1000 Iteration 500 1000 Response Length Sync Async [PITH_FULL_IMAGE:figures/full…

**Figure 12.** Figure 12: Reward and average response length comparison for the proposed asynchronous RL workflow and vanilla synchronous RL workflow. classify post-training frameworks into two paradigms: taskcollocated and task-separated, based on how reinforcement learning tasks map to phy…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
cs.DC 2026-06 unverdicted novelty 6.0 of 10

RolloutPipe uses complete-group pipelining and frontier-group dispatch to overlap rollout and training in disaggregated on-policy RLVR while preserving correctness, cutting end-to-end time by 30.7-42.3% versus Slime.
DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Training
cs.AI 2026-06 conditional novelty 6.0 of 10

Dynamically reassigning GPUs between rollout and training during LLM post-training cuts pipeline bubbles, improving measured throughput by 66.5% over the best static split in the paper's 8-GPU test.
StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training
cs.DC 2026-01 conditional novelty 6.0 of 10

StaleFlow enforces per-trajectory staleness bounds and coordinates rollout data routing, achieving 1.42–2.68x higher RL post-training throughput with comparable convergence.
QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization
cs.AI 2026-07 conditional novelty 5.0 of 10

QLPO resamples GRPO training groups to favor short correct and long incorrect responses, cutting reasoning length substantially while keeping accuracy roughly unchanged.

Reference graph

Works this paper leans on

47 extracted references · 14 canonical work pages · cited by 4 Pith papers

[1]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova
[2]

Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. 2025. AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Lan- guage Reasoning. arXiv preprint arXiv:2505.24298 (2025)

arXiv 2025
[3]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

arXiv 2024
[4]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. 11 Han, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforce- ment learning. arXiv preprint arXiv:2501.12948 (2025)

arXiv 2025
[5]

Jian Hu, Xibin Wu, Zilin Zhu, Weixun Wang, Dehao Zhang, Yu Cao, et al. 2024. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143 (2024)

arXiv 2024
[6]

Huawei. 2025. MindSpeed-RL. https://gitee.com/ascend/MindSpeed- RL

work page 2025
[7]

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Ben- jamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

arXiv 2020
[8]

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

Show all 47 references

[9]

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al
[10]

Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. 2025. DeepScaleR: Surpassing O1- Preview with a 1.5B Model by Scaling RL. https://pretty-radio- b75.notion.site/De...

2025
[11]

Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. 2018. Ray: A distributed framework for emerg- ing AI applications. In 13th USENIX symposium on operating systems desi...

2018
[12]

Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. 2023. Scaling data-constrained language models. Advances in Neural Information Processing Systems 36 (2023), 50358– 50376

2023
[13]

Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hos- seini, Rishabh Agarwal, and Aaron Courville. 2024. Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models. arXiv preprint arXiv:2410.18252 (2024)

2024 arXiv
[14]

NVIDIA. 2024. Data Flywheel. https://www.nvidia.com/en-us/glossar y/data-flywheel/

2024
[15]

NVIDIA. 2024. TensorRT-LLM. https://nvidia.github.io/TensorRT- LLM/

2024
[16]

OpenAI. 2024. Learning to reason with LLMs. https://openai.com/ind ex/learning-to-reason-with-llms/

2024
[17]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wain- wright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al . 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems...

2022
[18]

Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot. In 23rd USENIX Conference on File and Storage Te...

2025
[19]

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al
[20]

Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. 2024. Group robust preference optimization in reward-free rlhf. Advances in Neural Information Processing Systems 37 (2024), 37100–37137

2024
[21]

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He
[22]

Redis. 2025. Redis. https://redis.io/

2025
[23]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

2017 arXiv
[24]

ByteDance Seed, Yufeng Yuan, Yu Yue, Mingxuan Wang, Xiaochen Zuo, Jiaze Chen, Lin Yan, Wenyuan Xu, Chi Zhang, Xin Liu, et al
[25]

Failures

SemiAnalysis. 2024. Scaling Laws – O1 Pro Architecture, Reasoning Training Infrastructure, Orion and Claude 3.5 Opus “Failures”. https: //semianalysis.com/2024/12/11/scaling-laws-o1-pro-architecture- reasoning-training-infrastructure-orion-and-claude-3-5-opus- failures/

2024
[26]

Gerald Shen, Zhilin Wang, Olivier Delalleau, Jiaqi Zeng, Yi Dong, Daniel Egert, Shengyang Sun, Jimmy Zhang, Sahil Jain, Ali Taghibakhshi, et al. 2024. Nemo-Aligner: Scalable toolkit for efficient model alignment. arXiv preprint arXiv:2405.01481 (2024)

2024 arXiv
[27]

Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hy- bridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409.19256 (2024)

2024 arXiv
[28]

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGres- ley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Train- ing multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)

2019 arXiv
[29]

Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al
[30]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. At- tention is all you need. Advances in neural information processing systems 30 (2017)

2017
[31]

Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. 2024. Position: Will we run out of data? Limits of LLM scaling based on human-generated data. In Forty-first International Conference on Machine Learning

2024
[32]

Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020. TRL: Transformer Reinforcement Learning. https://github.com/huggingface/trl

2020
[33]

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021)

2021 arXiv
[34]

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 (2024)

2024 arXiv
[35]

Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajb- handari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, et al. 2023. DeepSpeed-Chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales. arXiv prep...

2023 arXiv
[36]

5: Scaling reinforcement learning with llms

Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599 (2025)

2025 arXiv
[37]

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 1, 2 (2023)

2023 arXiv
[38]

Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, et al
[39]

Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, and Xin Jin. 2025. Optimizing RLHF Training for Large Language Models with Stage Fusion. arXiv:2409.13221 [cs.LG] https://arxiv.org/abs/24 09.13221 13

2025 arXiv
[43]

Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. 2025. DAPO: An open-source llm reinforcement learning system at scale. 12 AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Tra...

2025 arXiv
[46]

StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation.arXiv preprint arXiv:2504.15930 (2025)

2025 arXiv
[2018]

Improving language understanding by generative pre-training. (2018)

2018
[2019]

In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers)

Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) . 4171–4186

2019
[2020]

In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining

DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 3505–3506
[2023]

InProceedings of the 29th Symposium on Operating Systems Principles

Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles. 611–626
[2024]

arXiv preprint arXiv:2412.19437 (2024)

Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

2024 arXiv
[2025]

arXiv preprint arXiv:2504.13914 (2025)

Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning. arXiv preprint arXiv:2504.13914 (2025)

2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

[2] [2]

Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. 2025. AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Lan- guage Reasoning. arXiv preprint arXiv:2505.24298 (2025)

arXiv 2025

[3] [3]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

arXiv 2024

[4] [4]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. 11 Han, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforce- ment learning. arXiv preprint arXiv:2501.12948 (2025)

arXiv 2025

[5] [5]

Jian Hu, Xibin Wu, Zilin Zhu, Weixun Wang, Dehao Zhang, Yu Cao, et al. 2024. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143 (2024)

arXiv 2024

[6] [6]

Huawei. 2025. MindSpeed-RL. https://gitee.com/ascend/MindSpeed- RL

work page 2025

[7] [7]

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Ben- jamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

arXiv 2020

[8] [8]

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica