Pith. sign in

REVIEW 9 cited by

Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.18252 v3 pith:2TAQZXOZ submitted 2024-10-23 cs.LG cs.AIcs.CL

Michael Noukhovitch , Shengyi Huang , Sophie Xhonneux , Arian Hosseini , Rishabh Agarwal , Aaron Courville This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords asynchronousrlhftrainingfasterlearningmodelwhileoff-policy
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The dominant paradigm for RLHF is online and on-policy RL: synchronously generating from the large language model (LLM) policy, labelling with a reward model, and learning using feedback on the LLM's own outputs. While performant, this paradigm is computationally inefficient. Inspired by classical deep RL literature, we propose separating generation and learning in RLHF. This enables asynchronous generation of new samples while simultaneously training on old samples, leading to faster training and more compute-optimal scaling. However, asynchronous training relies on an underexplored regime, online but off-policy RLHF: learning on samples from previous iterations of our model which give a worse training signal. We tackle the fundamental challenge in this regime: how much off-policyness can we tolerate for asynchronous training to speed up learning but maintain performance? Among several RLHF algorithms we test, online DPO is found to be most robust to off-policy data, and robustness increases with the scale of the policy model. We study further compute optimizations for asynchronous RLHF but find that they come at a performance cost, giving rise to a trade-off. We verify the scalability of asynchronous RLHF by training a general-purpose chatbot from LLaMA 3.1 8B on an instruction-following task ~40% faster than a synchronous run while matching final performance. Finally, we extend our results to math and reasoning to demonstrate asynchronous RL can finetune Rho 1B on GSM8k ~70% faster while matching synchronous accuracy.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Trees from Marginals: Autoregressive drafting with factorized priors

    cs.LG 2026-07 accept novelty 7.0 of 10

    Weaver restores conditional dependencies on top-K factorized marginals to build high-acceptance draft trees, plus a fused GDN tree-verify kernel, yielding 4.37× AR speedup and 24.7% over DFlash.

  2. Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    SAO stabilizes asynchronous RL for LLMs by replacing group-wise sampling with single-rollout updates, token-level importance sampling, and targeted value-model training, outperforming GRPO on reasoning and coding benchmarks.

  3. StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training

    cs.DC 2026-01 conditional novelty 6.0 of 10

    StaleFlow enforces per-trajectory staleness bounds and coordinates rollout data routing, achieving 1.42–2.68x higher RL post-training throughput with comparable convergence.

  4. Controllably Efficient Language Models

    cs.LG 2025-11 conditional novelty 6.0 of 10

    A single transformer variant can compress past context into chunk summaries and use chunk size as a test-time knob to trade quality against speed and memory, outperforming many efficient baselines on recall benchmarks.

  5. Magistral

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Pure RL alone, without distilled reasoning traces, turned Mistral's base models into strong reasoning models on math and coding benchmarks.

  6. Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts

    cs.AI 2025-06 conditional novelty 6.0 of 10

    An online pre-rollout filter that predicts and skips zero-variance prompts from reward dynamics reduces GRPO rollout time by up to 2.4x with comparable math reasoning accuracy.

  7. ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning

    cs.LG 2026-02 conditional novelty 5.0 of 10

    Distributed rollout workers under a bounded-staleness budget can keep a centralized learner saturated and cut LLM post-training cost by roughly a third at matched reward.

  8. AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training

    cs.LG 2025-07 conditional novelty 4.0 of 10

    AsyncFlow combines a distributed streaming data queue with delayed parameter updates to improve RL post-training throughput by 1.59x on average over verl on Ascend clusters.

  9. RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models

    cs.CL 2025-06 reject novelty 3.0 of 10

    RACE-Align generates preference pairs from RAG-grounded chain-of-thought answers and applies DPO to align a 1.7B model, showing improved reasoning scores in TCM QA but lacking statistical support.

Pith tools