Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LlamaRL proves that decoupling the generator and trainer into asynchronous GPU groups strictly outperforms any co-located synchronous RL framework under the same hardware, with up to 10.7x speedup at 405B.

desk verdict Solid systems paper with real large-scale speedups, but the 'strictly faster than any synchronous framework' theorem only holds against a co-located baseline that can't phase-swap memory; scope the claim and it's a genuinely useful contribution. read the letter →

arxiv 2505.24034 v2 pith:Q3D3J6G6 submitted 2025-05-29 cs.LG cs.AI

Bo Wu , Sid Wang , Yunhao Tang , Jia Ding , Eryk Helenowski , Liang Tan , Tengyu Xu , Tushar Gowda
show 6 more authors
Zhengxing Chen Chen Zhu Xiaocheng Tang Yundi Qian Beibei Zhu Rui Hou
This is my paper
classification cs.LGcs.AI
keywords reinforcementlearninglargelanguagemodelsasynchronoustrainingoff-policyimportancesamplingdistributedsystemsmodelparallelismLLMpost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that the usual way of running RL post-training on large language models—co-locating response generation and policy updates on the same GPUs and alternating between them—leaves a large amount of speed on the table. It presents LlamaRL, a distributed asynchronous framework in which a generator executor and a trainer executor run on separate GPU groups with independent parallelisms, batch sizes, and precisions, synchronized by GPU-native direct memory transfers. A formal theorem claims that under the same hardware and memory budget this asynchronous architecture is strictly faster than any co-located synchronous configuration, and experiments report 2.52x, 3.98x, and 10.7x per-step speedups at 8B, 70B, and 405B with comparable math-reasoning quality. If correct, the result makes large-scale RL fine-tuning much cheaper and identifies asynchrony—not just faster kernels—as the main lever for scaling LLM post-training.

What carries the argument

The load-bearing identity is the RL step-time comparison: $T_{\mathrm{baseline}} = \frac{B_0}{G_0} m(\eta_t + \eta_g)$ versus $T_{\mathrm{LlamaRL}} = \frac{B_0}{G_0}\max(\eta_t m_t / \theta, \eta_g m_g / (1-\theta))$, where $\eta_t,\eta_g$ are per-sample processing times, $m_t,m_g$ are model-parallel degrees, and $\theta$ is the trainer's GPU fraction. The proof's premise is Assumption 7.1, that per-sample processing time decreases with batch size. Three engineering mechanisms carry the result: a single-controller executor architecture with typed communication channels; Distributed Direct Memory Access (DDMA), which synchronizes sharded weights between training and generation across NVLink and InfiniBand in about two seconds for terabyte-scale models; and AIPO (Asynchronous Importance weighted Policy Optimization), a one-sided clipped importance-sampling correction $\min(\pi(y_t)/\mu(y_t), \rho)$ that stabilizes off-policy learning.

What would settle it

Measure per-step time on the same 1024-H100, 405B setup using a synchronous framework that is allowed to give the generator its own fp8 group and its own model-parallel degree while still alternating logically between training and generation; if that configuration's step time is not strictly larger than LlamaRL's 59.5 seconds, the theorem's universal claim fails. Separately, measure per-sample generation and training time at 405B across the full batch-size range used in the experiments and check whether both keep decreasing with batch size; if either curve flattens or rises, Assumption 7.1 is violated and the proof's premise collapses.

Watch

Extended reading notes

Core claim

The central discovery is Theorem 7.5: given the same hardware budget and memory constraints, LlamaRL can be configured so that its RL step time $T_{\mathrm{LlamaRL}}$ is strictly smaller than the minimum possible step time $T_{\mathrm{baseline}}$ of a co-located synchronous RL framework. The theorem frames speed as a constrained optimization problem, with the synchronous step time proportional to $m(\eta_t + \eta_g)$ under one shared per-GPU memory constraint, and the asynchronous step time proportional to $\max(\eta_t m_t / \theta, \eta_g m_g / (1-\theta))$ under two independent constraints. Decoupling memory frees each side to use larger batches and lower model-parallel degrees, so the slower of the two components replaces their sum. Empirically, on 405B with 1024 H100 GPUs and a fixed global batch of 2048, per-step time drops from 635.8 to 59.5 seconds while MATH-500, MATH, and GSM8K scores stay on par with the synchronous baseline.

Load-bearing premise

The strict-speedup theorem holds only if the comparison baseline is required to keep generator and trainer co-located with one shared model-parallel split, and if larger batches keep lowering per-sample processing time in both phases; both are assumptions rather than universal laws.

Editorial extensions

If this is right

  • At 405B scale, LlamaRL lowers per-step RL time from 635.8 seconds to 59.5 seconds on the same 1024 H100 GPUs, a 10.7x throughput gain with the same global batch size.
  • The speedup over the synchronous baseline grows with model size along a convex curve in log scale, so the framework's advantage should widen for models beyond 405B.
  • DDMA reduces weight synchronization for terabyte-scale models to roughly two seconds, removing the 900-second-class weight-reload bottleneck observed for 405B in earlier co-located frameworks.
  • Because generator and trainer can use different parallelism and precision, generation can run at fp8 with low model parallelism while the trainer stays at bf16 with higher parallelism, balancing the global pipeline; decoupling alone still gives a 2.6x gain at 405B before quantization.
  • Off-policy asynchronous training with AIPO preserves model quality: 8B LlamaRL matches the synchronous baseline on MATH and GSM8K while showing consistent gains on MATH-500.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Read tightly, the proof covers the co-located synchronous baseline with a single shared model-parallel degree; whether the strict inequality extends to distributed synchronous frameworks that already decouple trainer and generator is a separate question, since those fall outside the theorem's baseline definition.
  • A direct experimental extension would repeat the 405B comparison with the synchronous baseline given the same fp8 generator and independent model-parallel choices; the gap between that result and the 10.7x figure would isolate the contribution of asynchrony from the contribution of quantization.
  • The one-sided clipped importance weighting in AIPO may generalize to any pipeline where the behavior policy lags the learner; combining it with two-sided PPO clipping offers a variance-stability trade-off worth testing in other off-policy setups.
  • DDMA's linear scalability across thousands of GPUs is the next point to stress-test: at multi-terabyte weights and mixed inter-node topologies, the two-second synchronization claim should be re-measured, since the theorem treats inter-device synchronization time as negligible.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript presents LlamaRL, a fully distributed asynchronous RL framework for large-scale LLM post-training. The framework uses a single-controller executor architecture on native PyTorch, distributed direct memory access (DDMA) for weight synchronization, decoupled parallelism and quantization between generator and trainer, and an importance-weighted off-policy algorithm called AIPO. The paper claims a theoretical result (Theorem 7.5) that, under a monotone batch-size scaling assumption, LlamaRL is strictly faster than any possible configuration of a traditional synchronous RL framework given the same GPU budget and memory constraints. It reports 2.52x, 3.98x, and 10.7x RL step-time speedups over a DeepSpeed-Chat-like co-located synchronous baseline for 8B, 70B, and 405B Llama 3.1 models, with model quality evaluated only at 8B.

Significance. If the claims held in full, this would be a substantial systems contribution: it demonstrates asynchronous RL training at 405B scale, introduces a plausible off-policy correction with a stability ablation, and provides a tractable memory-based step-time model. The DDMA weight-synchronization measurements in Table 4 and the AIPO stability results in Figure 8 are practically valuable, and the proof of Theorem 7.5 is explicit and assumption-driven rather than circular. However, the headline theoretical guarantee is currently overstated: the proof's baseline excludes phase-swap synchronous execution, so the strict inequality in Theorem 7.5 does not extend to 'any possible configuration' of a synchronous framework. The empirical scaling claim is also confounded by quantization asymmetry at 405B. These issues do not negate the engineering results, but they require substantial revision of the theorem's scope and of the scaling narrative.

major comments (4)
  1. [Section 7, Eq. (6)] The strict speedup is an artifact of the chosen baseline definition. The baseline memory constraint in Eq. (6), ((4W0 + At bt) + (W0 + Kg bg)) / m <= M0, requires trainer and generator memory footprints to be simultaneously co-resident under a single shared parallel degree m. A synchronous framework may instead run generation and training sequentially and free the opposite phase's state: during generation, optimizer states and gradients are not needed, and during training, the KV cache is not needed. Such a phase-swap baseline has separate memory constraints (4W0 + At bt) / mt <= M0 and (W0 + Kg bg) / mg <= M0 with independent mt and mg. Under the paper's own Assumption 7.1, its optimal step time is (B0/G0) * [min_bt (4W0 + At bt) ηt / M0 + min_bg (W0 + Kg bg) ηg / M0], which is exactly the quantity Eq. (11) identifies as the LlamaRL optimum T**_t + T**_g. The inequality in Theorem 7.5 therefore becomes equality against this natural synchronous baseline. Consequently, Theorem 7.1's claim of strict speedup over 'any possible configuration of a traditional synchronous RL framework' is unsupported and, under this definition of synchronous execution, false. Please restrict the theorem to baselines that co-reside trainer and generator state, or justify why phase-wise offloading is excluded, and revise the informal statements and the conclusion accordingly.
  2. [Section 7, Eq. (7) and Footnote 1] The optimization in problem (7) treats mt and mg as real-valued and omits the GPU-budget constraints mt <= θ G0 and mg <= (1−θ) G0; Footnote 1 acknowledges that the m <= G0 constraint is omitted and asserts it is satisfied under sufficient GPU budget, but no formal condition is stated or verified. Lemma B.2 then sets mt and mg exactly to memory ratios such as (4W0 + At bt) / M0, which may be non-integral or may sum to more than G0. Because the theorem's premise is 'same hardware budget,' the proof must either include these constraints or state an explicit sufficient-GPU-budget assumption and verify it for the configurations used in Table 3. Without this, the formal guarantee can select infeasible configurations and the strict speedup is not established for the stated premise.
  3. [Section 8.2, Table 3] The claim that the efficiency advantage 'continues to grow with increasing model scale' is not cleanly supported by the reported experiments. At 405B, the best LlamaRL row (59.5 seconds) uses an fp8 generator with mp=8, while the baseline row (635.8 seconds) uses bf16 with mp=64; the text acknowledges that the gain 'partly benefits from fp8 quantization' but then attributes the scaling trend primarily to parallelism flexibility. To support the scaling claim, the comparison should include a 405B LlamaRL run without fp8, an fp8-capable baseline, or an explicit statement that the reported speedup is an end-to-end system comparison with mixed optimizations. Additionally, Assumption 7.1 is empirically validated only at 70B in Figure 5, so the theoretical basis for extrapolating the monotone trend to 405B and beyond is not directly evidenced.
  4. [Section 8.3, Remark] Quality parity is established only for the 8B model; the remark states that the 70B model does not obtain significant improvements by training on the MATH training set, and no 405B quality results are reported. The conclusion's statement that asynchronous training preserves 'training stability and final performance' is therefore stronger than the evidence presented. Please add large-scale quality results or explicitly qualify the claim to the 8B setting.
minor comments (4)
  1. [Throughout] There are numerous typos and grammatical slips: 'oevrall' in Section 4.2, 'co-located stragety' and 'stacking together' in Section 4.3, 'incoorperated' in Section 8.1, 'minimium' in the proof of Theorem 7.5, and 'we conclude Theorem 7.5' should be 'we prove Theorem 7.5'. A careful copyedit would improve readability.
  2. [References] Several references are duplicated or conflated: Bai et al. 2022a and 2022b are identical, Lightman et al. 2023a and 2023b are identical, Rajbhandari et al. 2019a and 2019b are identical, and Llama Team 2024 duplicates Dubey et al. 2024. These should be consolidated.
  3. [Section 5.2] The DDMA section would benefit from a description of the synchronization protocol's consistency semantics: it is not stated whether the trainer broadcasts complete weights or only shards, how often staleness is bounded, and whether the 2-second figure includes the full end-to-end path from updated optimizer state to generator-ready weights.
  4. [Section 8.3, Figure 6] The MATH-500 improvement for LlamaRL is noted as potentially statistical variation, but no error bars or multiple-seed results are reported for any quality comparison; adding at least multiple-seed or multiple-eval statistics would strengthen the quality-parity claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 7.5 is a derived comparison under explicit assumptions; its 'any configuration' wording overreaches but is a scoping issue, not a circular reduction.

full rationale

The central algorithmic component, AIPO, is an importance-sampling-weighted policy gradient explicitly connected to IMPALA and other prior off-policy methods; it is not fitted to the empirical speedup and does not assume its own conclusion. Theorem 7.5 is proven from stated definitions (Definition 7.4), Assumption 7.1, and the memory constraints in Eqs. (4)-(7); the proof's inequality chain in Eq. (11) does real mathematical work by splitting the shared memory constraint and independently optimizing trainer and generator. The strict speedup is established only against the co-located baseline with mt = mg = m that the paper defines in Eq. (6). The informal Theorem 7.1 claims 'any possible configuration of a traditional synchronous RL framework', which is broader than the model in Eq. (6); a phase-swapping synchronous baseline with separate peak-memory constraints could tie the asynchronous optimum. That is an overstatement of the theorem's scope, not a circular derivation. Similarly, the empirical speedup is measured against an external DeepSpeed-Chat-like baseline, and where the 405B gain benefits from fp8 generator quantization, the paper explicitly discloses this and attributes the main gain to parallelism decoupling. Assumption 7.1 is validated at 70B and extrapolated to 405B, which is an empirical-support gap rather than circular reasoning. No load-bearing self-citation chain is present; references to the Llama team's own model release are contextual, not the basis of the derivation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the monotone batch-size scaling assumption, the memory model of Table 2, the neglect of communication cost, continuous relaxation in the optimization, and a baseline restricted to co-located GPU placement. These are mostly stated domain assumptions; none is an invented entity. The AIPO clipping constant rho is a hand-tuned hyperparameter.

free parameters (3)
  • Importance-weight clipping constant rho = rho in [2,10], exact value not reported
    Selected from early ablations to stabilize async training (Section 6 and Appendix A).
  • KL regularization coefficient lambda_KL = not reported
    Introduced in Section 6 as controlling KL penalty; no value or sweep given.
  • Learning rate = 2e-7
    Fixed across all experiments (Section 8.1); affects the quality comparison.
assumptions (5)
  • domain assumption Assumption 7.1: per-sample training and generation times decrease monotonically with batch size.
    Validated empirically at 70B in Figure 5, not at 405B; used to prove Lemmas B.1 to B.3.
  • domain assumption Memory footprint model in Table 2 (model, Adam states, gradients, activations for trainer; model and KV cache for generator) dominates per-GPU memory.
    Used to define memory constraints in Eq. (4) to (7).
  • domain assumption Inter-device communication overhead is negligible.
    Stated in Section 7; DDMA sync for 405B is 2.31s, about 4 percent of the 59.5s step, so it is not strictly negligible in the theorem.
  • ad hoc to paper Continuous relaxation: batch sizes and parallel degrees are treated as real numbers and the m <= G0 GPU budget constraint is omitted.
    Footnote on page 16 acknowledges omitting m <= G0; this makes the optimization tractable but not a formal proof for discrete systems.
  • domain assumption Baseline comparison class is restricted to co-located generator and trainer with a shared model-parallel degree m.
    Defines T_baseline in Eq. (2) and Eq. (6); excludes distributed synchronous frameworks such as OpenRLHF and HybridFlow.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training." pith.science (2026). https://pith.science/paper/Q3D3J6G6

@misc{pith2026250524034,
  author       = {Pith},
  title        = {Pith review of: LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q3D3J6G6}},
  note         = {Machine review of arXiv:2505.24034}
}
read the original abstract

Reinforcement Learning (RL) has become the most effective post-training approach for improving the capabilities of Large Language Models (LLMs). In practice, because of the high demands on latency and memory, it is particularly challenging to develop an efficient RL framework that reliably manages policy models with hundreds to thousands of billions of parameters. In this paper, we present LlamaRL, a fully distributed, asynchronous RL framework optimized for efficient training of large-scale LLMs with various model sizes (8B, 70B, and 405B parameters) on GPU clusters ranging from a handful to thousands of devices. LlamaRL introduces a streamlined, single-controller architecture built entirely on native PyTorch, enabling modularity, ease of use, and seamless scalability to thousands of GPUs. We also provide a theoretical analysis of LlamaRL's efficiency, including a formal proof that its asynchronous design leads to strict RL speed-up. Empirically during the Llama 3 post-training, by leveraging best practices such as colocated model offloading, asynchronous off-policy training, and distributed direct memory access for weight synchronization, LlamaRL achieves significant efficiency gains -- up to 10.7x speed-up compared to DeepSpeed-Chat-like systems on a 405B-parameter policy model. Furthermore, the efficiency advantage continues to grow with increasing model scale, demonstrating the framework's suitability for future large-scale RL training.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training

    cs.DC 2026-01 conditional novelty 6.0 of 10

    StaleFlow enforces per-trajectory staleness bounds and coordinates rollout data routing, achieving 1.42–2.68x higher RL post-training throughput with comparable convergence.

  2. Magistral

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Pure RL alone, without distilled reasoning traces, turned Mistral's base models into strong reasoning models on math and coding benchmarks.

  3. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

74 extracted references · 32 canonical work pages · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  3. [3]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  4. [4]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  5. [5]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page

  6. [6]

    Ahmadian, A., Cremer, C., Gall \'e , M., Fadaee, M., Kreutzer, J., Pietquin, O., \"U st \"u n, A., and Hooker, S. (2024). Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740

    arXiv 2024

  7. [7]

    Claude: Training helpful and harmless ai assistants

    Anthropic (2023). Claude: Training helpful and harmless ai assistants. Anthropic AI Blog . URL: https://www.anthropic.com/index/2023/Claude

    work page 2023

  8. [9]

    Bai, Y., Jones, A., Ndousse, K., and et al. (2022b). Training a helpful and harmless assistant with rlhf. arXiv preprint arXiv:2204.05862

    arXiv 2022

Show all 74 references
  1. [10]

    Berner, C., Brockman, G., Chan, B., Cheung, V., D e biak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. (2019). Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680

    2019 arXiv

  2. [11]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems , 33:1877--1901

    2020

  3. [12]

    Chilimbi, T., Suzue, Y., Apacible, J., and Kalyanaraman, K. (2014). Project Adam : Building an efficient and scalable deep learning training system. In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI) , pages 571--582

    2014

  4. [13]

    Christiano, P., Leike, J., Brown, T., and et al. (2017). Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS)

    2017

  5. [14]

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. ArXiv , abs/2110.14168

    2021 arXiv

  6. [15]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    DeepSeek-AI (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  7. [16]

    Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. (2023). Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767

    2023 arXiv

  8. [18]

    El-Kishky, A., Wei, A., Saraiva, A., Minaev, B., Selsam, D., Dohan, D., Song, F., Lightman, H., Clavera, I., Pachocki, J., et al. (2025). Competitive programming with large reasoning models. arXiv preprint arXiv:2502.06807

    2025 arXiv

  9. [19]

    Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. (2018). Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning , pag...

    2018

  10. [20]

    Gehring, J., Zheng, K., Copet, J., Mella, V., Cohen, T., and Synnaeve, G. (2024). Rlef: Grounding code llms in execution feedback with reinforcement learning. arXiv preprint arXiv:2410.02089

    2024 arXiv

  11. [21]

    Bard: Conversational ai by google

    Google (2023). Bard: Conversational ai by google. Google AI Blog . URL: https://blog.google/technology/ai/bard-google-ai/

    2023

  12. [22]

    Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., Dias, R., Vallone, A., Ren, H., Wei, J., et al

    Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., Dias, R., Vallone, A., Ren, H., Wei, J., et al. (2024). Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339

    2024 arXiv

  13. [23]

    X., and Steinhardt, J

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D. X., and Steinhardt, J. (2021). Measuring mathematical problem solving with the math dataset. ArXiv , abs/2103.03874

    2021 arXiv

  14. [24]

    Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. (2017). Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409

    2017 arXiv

  15. [25]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.15556

    2022 arXiv

  16. [26]

    Hu, J., Wu, X., Wang, W., Xianyu, Zhang, D., and Cao, Y. (2024). Openrlhf: An easy-to-use, scalable and high-performance rlhf framework

    2024

  17. [27]

    Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al. (2024). Openai o1 system card. arXiv preprint arXiv:2412.16720

    2024 arXiv

  18. [28]

    and Abbeel, P

    Jie, T. and Abbeel, P. (2010). On a connection between importance sampling and the likelihood ratio policy gradient. Advances in Neural Information Processing Systems , 23

    2010

  19. [29]

    and Langford, J

    Kakade, S. and Langford, J. (2002). Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference on machine learning , pages 267--274

    2002

  20. [30]

    Kaplan, J., McCandlish, S., Henighan, T., and et al. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  21. [31]

    Kapturowski, S., Ostrovski, G., Quan, J., Munos, R., and Dabney, W. (2018). Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations

    2018

  22. [32]

    Kazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., and Roux, N. L. (2024). Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679

    2024 arXiv

  23. [33]

    Kimi k1.5: Scaling reinforcement learning with llms

    Kimi Team (2025). Kimi k1.5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599 . Version 2, revised on March 5, 2025

    2025 arXiv

  24. [34]

    Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  25. [35]

    D., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L

    Lambert, N., Morrison, J. D., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, X., Gu, Y., Malik, S., Graf, V., Hwang, J. D., Yang, J., Bras, R. L., Tafjord, O., Wilhelm, C., Soldaini, L., Smith, N. A., Wang, Y., Dasigi, P., and Haji...

    2024 arXiv

  26. [36]

    G., Park, J

    Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A., Josifovski, V., and Long, J. (2014). Scaling distributed machine learning with the parameter server. In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI) , pages 583--598

    2014

  27. [37]

    Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022). Competition-level code generation with alphacode. Science , 378(6624):1092--1097

    2022

  28. [38]

    Li, Z., Xu, T., Zhang, Y., Lin, Z., Yu, Y., Sun, R., and Luo, Z.-Q. (2024). Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models

    2024

  29. [40]

    Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. (2023b). Let's verify step by step. ArXiv , abs/2305.20050

    2023 arXiv

  30. [41]

    The llama 3 herd of models

    Llama Team (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  31. [42]

    and Hutter, F

    Loshchilov, I. and Hutter, F. (2017). Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  32. [43]

    P., Paprocki, M., C ert\' i k, O., Kirpichev, S

    Meurer, A., Smith, C. P., Paprocki, M., C ert\' i k, O., Kirpichev, S. B., Rocklin, M., Kumar, A., Ivanov, S., Moore, J. K., Singh, S., Rathnayake, T., Vig, S., Granger, B. E., Muller, R. P., Bonazzi, F., Gupta, H., Vats, S., Johansson, F., Pedregosa, F., Curry, M. J., Terrel,...

    2017

  33. [44]

    P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K

    Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928--1937. PmLR

    2016

  34. [45]

    I., and Stoica, I

    Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Elibol, M., Yang, Z., Paul, W., Jordan, M. I., and Stoica, I. (2018). Ray: A Distributed Framework for Emerging AI Applications . In Proceedings of the 13th USENIX Symposium on Operating Systems Design and ...

    2018

  35. [46]

    Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. (2016). Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems , 29

    2016

  36. [47]

    B., Singh, V., Lin, M., Gimelshein, N., Desmaison, A., and Yang, E

    Nguyen, V., Carilli, M., Eryilmaz, S. B., Singh, V., Lin, M., Gimelshein, N., Desmaison, A., and Yang, E. (2021). Accelerating PyTorch with CUDA graphs. https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/

    2021

  37. [48]

    Noukhovitch, M., Huang, S., Xhonneux, S., Hosseini, A., Agarwal, R., and Courville, A. (2024). Asynchronous rlhf: Faster and more efficient off-policy rl for language models

    2024

  38. [49]

    Gpt-4 technical report

    OpenAI (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    2023 arXiv

  39. [50]

    Ouyang, L., Wu, J., Jiang, X., and et al. (2022). Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155

    2022 arXiv

  40. [51]

    and et al

    Potluri, S. and et al. (2013). GPU Direct RDMA for infiniband on NVIDIA GPUs : A case study. In Proceedings of the 2013 IEEE International Symposium on Cluster, Cloud and Grid Computing (CCGrid) , pages 340--347

    2013

  41. [52]

    S., and Singh, S

    Precup, D., Sutton, R. S., and Singh, S. (2000). Eligibility traces for off-policy policy evaluation. In ICML , volume 2000, pages 759--766. Citeseer

    2000

  42. [53]

    Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. (2019a). Zero: Memory optimizations toward training trillion parameter models

    2019

  43. [54]

    Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. (2019b). Zero: Memory optimizations toward training trillion parameter models. SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1--16

    2019

  44. [55]

    Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. (2020). Zero: Memory optimization towards training trillion parameter models. Proceedings of Machine Learning and Systems

    2020

  45. [56]

    Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015). Trust region policy optimization. In International conference on machine learning , pages 1889--1897. PMLR

    2015

  46. [57]

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  47. [58]

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. (2024). Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  48. [59]

    S., Aithal, A., and Kuchaiev, O

    Shen, G., Wang, Z., Delalleau, O., Zeng, J., Dong, Y., Egert, D., Sun, S., Zhang, J., Jain, S., Taghibakhshi, A., Ausin, M. S., Aithal, A., and Kuchaiev, O. (2024). Nemo-aligner: Scalable toolkit for efficient model alignment

    2024

  49. [60]

    Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., and Wu, C. (2024). Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409.19256

    2024 arXiv

  50. [61]

    Shoeybi, M., Patwary, M. M. A., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. (2019). Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053

    2019 arXiv

  51. [62]

    Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2017a). Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815

    2017 arXiv

  52. [63]

    Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017b). Mastering the game of go without human knowledge. nature , 550(7676):354--359

    2017

  53. [64]

    Stiennon, N., Ouyang, L., Wu, J., and et al. (2020). Learning to summarize with human feedback. Advances in Neural Information Processing Systems (NeurIPS)

    2020

  54. [65]

    M., Hauth, A., Millican, K., et al

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023). Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  55. [66]

    Tian, Y., Ma, J., Gong, Q., Sengupta, S., Chen, Z., Pinkerton, J., and Zitnick, L. (2019). Elf opengo: An analysis and open reimplementation of alphazero. In International conference on machine learning , pages 6244--6253. PMLR

    2019

  56. [67]

    Touvron, H., Martin, L., Stone, K., and et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  57. [68]

    Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, I. (2022). Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275

    2022 arXiv

  58. [69]

    M., Dudzik, A., Huang, A., Georgiev, P., Powell, R., et al

    Vinyals, O., Babuschkin, I., Chung, J., Mathieu, M., Jaderberg, M., Czarnecki, W. M., Dudzik, A., Huang, A., Georgiev, P., Powell, R., et al. (2019). Alphastar: Mastering the real-time strategy game starcraft ii. DeepMind blog , 2:20

    2019

  59. [70]

    Wang, Z., Bapst, V., Heess, N., Mnih, V., Munos, R., Kavukcuoglu, K., and De Freitas, N. (2016). Sample efficient actor-critic with experience replay. arXiv preprint arXiv:1611.01224

    2016 arXiv

  60. [71]

    Wei, Y., Duchenne, O., Copet, J., Carbonneaux, Q., Zhang, L., Fried, D., Synnaeve, G., Singh, R., and Wang, S. I. (2025). Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18449

    2025 arXiv

  61. [72]

    Xiao, Y., Zhou, Z., Mao, F., Wu, W., Zhao, S., Ju, L., Liang, L., Zhang, X., and Zhou, J. (2023). An adaptive placement and parallelism framework for accelerating rlhf training. arXiv preprint arXiv:2312.11819

    2023 arXiv

  62. [73]

    A., Jin, D., Peng, K., Han, E., Nie, S., Zhu, C., Zhang, H., Zhou, W., Zeng, Z., He, Y., Mandyam, K., Talabzadeh, A., Khabsa, M., Cohen, G., Tian, Y., Ma, H., Wang, S., and Fang, H

    Xu, T., Helenowski, E., Sankararaman, K. A., Jin, D., Peng, K., Han, E., Nie, S., Zhu, C., Zhang, H., Zhou, W., Zeng, Z., He, Y., Mandyam, K., Talabzadeh, A., Khabsa, M., Cohen, G., Tian, Y., Ma, H., Wang, S., and Fang, H. (2024). The perfect blend: Redefining rlhf with mixtur...

    2024

  63. [74]

    Y., Ruwase, O., Rajbhandari, S., Wu, X., Awan, A

    Yao, Z., Aminabadi, R. Y., Ruwase, O., Rajbhandari, S., Wu, X., Awan, A. A., Rasley, J., Zhang, M., Li, C., Holmes, C., Zhou, Z., Wyatt, M., Smith, M., Kurilenko, L., Qin, H., Tanaka, M., Che, S., Song, S. L., and He, Y. (2023). Deepspeed-chat: Easy, fast and affordable rlhf t...

    2023

  64. [75]

    Zhang, B., Liu, Z., Cherry, C., and Firat, O. (2024). When scaling meets llm finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193

    2024 arXiv

  65. [76]

    Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., Desmaison, A., Balioglu, C., Damania, P., Nguyen, B., Chauhan, G., Hao, Y., Mathews, A., and Li, S. (2023). Pytorch fsdp: Experiences on scaling fully sharded data parallel

    2023

  66. [77]

    M., Stiennon, N., Wu, J., Brown, T

    Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2019). Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

    2019 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.