REVIEW 9 cited by
StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs). RL for LLMs involves two stages: generation and training. The LLM first generates samples online, which are then used to derive rewards for training. The conventional view holds that the colocated architecture, where the two stages share resources via temporal multiplexing, outperforms the disaggregated architecture, in which dedicated resources are assigned to each stage. However, in real-world deployments, we observe that the colocated architecture suffers from resource coupling, where the two stages are constrained to use the same resources. This coupling compromises the scalability and cost-efficiency of colocated RL in large-scale training. In contrast, the disaggregated architecture allows for flexible resource allocation, supports heterogeneous training setups, and facilitates cross-datacenter deployment. StreamRL is designed with disaggregation from first principles and fully unlocks its potential by addressing two types of performance bottlenecks in existing disaggregated RL frameworks: pipeline bubbles, caused by stage dependencies, and skewness bubbles, resulting from long-tail output length distributions. To address pipeline bubbles, StreamRL breaks the traditional stage boundary in synchronous RL algorithms through stream generation and achieves full overlapping in asynchronous RL. To address skewness bubbles, StreamRL employs an output-length ranker model to identify long-tail samples and reduces generation time via skewness-aware dispatching and scheduling. Experiments show that StreamRL improves throughput by up to 2.66x compared to existing state-of-the-art systems, and improves cost-effectiveness by up to 1.33x in a heterogeneous, cross-datacenter setting.
Forward citations
Cited by 9 Pith papers
-
Bidirectional Resource Scheduling for Disaggregated and Asynchronous RL Post-Training
BiDiRL raises disaggregated asynchronous LLM RL throughput up to 1.94× by hot-switching idle GPUs between rollout and training under a model-guided bidirectional scheduler.
-
StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training
StaleFlow enforces per-trajectory staleness bounds and coordinates rollout data routing, achieving 1.42–2.68x higher RL post-training throughput with comparable convergence.
-
QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization
QLPO resamples GRPO training groups to favor short correct and long incorrect responses, cutting reasoning length substantially while keeping accuracy roughly unchanged.
-
Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning
Molt is a lean, readable PyTorch-native RL training framework that reports statistically comparable step-time throughput to a Megatron-based stack, with the caveat that the head-to-head comparison measured throughput only.
-
ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning
Distributed rollout workers under a bounded-staleness budget can keep a centralized learner saturated and cut LLM post-training cost by roughly a third at matched reward.
-
Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library
ROLL is a distributed RL training library for LLMs with sample-level rollout scheduling and flexible device mapping, demonstrated on RLVR and agentic tasks but without baseline or scaling measurements.
-
DeepCEE: Efficient Cross-Region Model Distributed Training System under Heterogeneous GPUs and Networks
DeepCEE groups heterogeneous GPUs by network and compute speed, schedules a compact zero-bubble pipeline across regions, and adapts micro-batch sizes to network fluctuations, reporting 1.3-2.8x higher training through...
-
AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
AsyncFlow combines a distributed streaming data queue with delayed parameter updates to improve RL post-training throughput by 1.59x on average over verl on Ascend clusters.
-
Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Models
A GRPO decoding framework that cuts memory via micro-batched KV-cache reuse and improves decoding-round efficiency with predicted-length scheduling, at the cost of serialization.
Discussion (0). Continue with ORCID to comment.