Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ECHO-2 reduces the dollar cost of LLM reinforcement-learning post-training by moving rollout generation to cheap distributed inference workers, while keeping final reward comparable to centralized pipelines.

desk verdict Useful industrial-systems paper with a clean capacity rule and plausible cost savings, but the headline cost numbers don't reproduce from its own formulas and the staleness robustness is asserted rather than proven. read the letter →

arxiv 2602.02192 v5 pith:FQB5Z6HR submitted 2026-02-02 cs.LG cs.DC

Jingwei Song , Meng Chen , Jie Xiao , Qingnan Ren , Jiaqi Huang , Yangshen Deng , Chris Tong , Wanyi Chen
show 10 more authors
Suli Wang Zhisheng Chen Ziqian Bi Shuo Lu Yiqun Duan Xu Wang Rymon Yu Lynn Ai Eric Yang Tianyu Shi
This is my paper · ORCID
classification cs.LGcs.DC
keywords distributedreinforcementlearningLLMpost-trainingrolloutoffloadingboundedstalenessGRPOcostefficiencypeer-assistedbroadcastprovisioningrule
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ECHO-2 argues that the dominant cost of LLM reinforcement-learning post-training — generating rollouts — can be moved off expensive data-center GPUs onto cheap, geographically scattered inference machines, as long as the learner tolerates a bounded amount of policy staleness. The paper makes this precise with an overlap-based capacity model: a single inequality relates per-step training time, dissemination latency, batch size, and aggregate worker throughput, and tells a practitioner exactly how much cheap rollout capacity is needed to keep the central learner continuously busy. ECHO-2 realizes that rule with two mechanisms: peer-assisted pipelined broadcast, which keeps wide-area model dissemination from becoming a bottleneck, and cost-aware worker activation, which hires the cheapest workers that meet the capacity target. In experiments on GRPO post-training of 4B and 8B models, the framework cuts cumulative dollar cost by roughly a third while matching the reward of strong centralized baselines — provided the staleness budget S stays within a safe range (about 6 or less in these experiments). The broader point is that bounded staleness can be treated as a first-class control parameter that trades rollout cost against training stability, not merely as an artifact to be minimized.

What carries the argument

The load-bearing object is the overlap condition: κT_train ≥ T_bcast + κR/Σμ_i, relating the snapshot publication period κ (default κ = S−1), per-update training time T_train, learner-visible dissemination latency T_bcast, rollouts per update R, and per-worker rollout throughput μ_i. It collapses a heterogeneous worker pool into one measurable aggregate-throughput requirement; satisfying it keeps the learner from idling. Three mechanisms implement it: bounded staleness (replay buffer discards trajectories older than the budget S), peer-assisted pipelined broadcast (striped store-and-forward chains reduce T_bcast to near the unlimited-bandwidth ideal), and cost-aware activation (greedily sele

What would settle it

Train a 4B model with GRPO on a task with dense, non-verifiable rewards (for example, a learned reward model for dialogue) under ECHO-2 with S = 3, and compare final reward to a synchronous baseline. A reward gap much larger than the roughly 5% band reported on math benchmarks would falsify the claim that bounded staleness preserves RL quality while cutting cost.

Watch

Extended reading notes

Core claim

ECHO-2 claims that RL post-training can be restructured as centralized learning plus distributed rollouts: a stable cluster runs the policy optimizer while a heterogeneous fleet of cheap remote workers generates rewarded trajectories over wide-area networks. The enabling assumption is that GRPO-style LLM objectives stay stable when the learner consumes rollouts from a policy up to S training steps old; ECHO-2 turns S into a user-specified staleness budget and publishes snapshots every κ = S−1 updates. The key formal result is the overlap condition κT_train ≥ T_bcast + κR/Σμ_i, a provisioning rule for keeping the learner saturated. Around it sit versioned bounded-staleness replay buffers, pee

Load-bearing premise

The entire cost-saving scheme rests on GRPO-style LLM post-training tolerating bounded policy staleness up to budget S without degrading reward — the paper verifies this empirically for math reasoning tasks but offers no formal guarantee that the safe staleness range generalizes to other tasks and reward signals.

Editorial extensions

If this is right

  • Practitioners can compute how many cheap workers are needed to saturate a learner before launching a run, turning provisioning from guesswork into arithmetic.
  • With a staleness budget of S ≤ 6, distributed rollouts preserve GRPO reward within a few percent of the synchronous baseline on math reasoning, so the cost saving does not come at the price of quality.
  • Peer-assisted broadcast keeps dissemination latency nearly flat as the worker fleet grows, even under capped uplink bandwidth, so the design scales to larger pools without linear broadcast-time growth.
  • Because only the task adapter changes between workloads, adding a new RL post-training task (such as a game environment) requires swapping dataset and reward logic rather than touching scheduling or infrastructure code.
  • The overlap condition also predicts where learner bubbles disappear, giving a diagnostic for under- or over-provisioned rollout pools.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The safe staleness range is likely task-dependent: on dense-reward or hard-to-verify tasks, the same S may degrade quality much faster, so the cost advantage narrows. Measuring per-task staleness tolerance is a natural next step.
  • The same overlap inequality extends beyond LLM RL to any pipeline pairing a central consumer with heterogeneous producers under bounded lag — synthetic-data generation, evaluation pipelines, or data curation could use the same rule to decide how many cheap workers to hire.
  • Running the scheduler adaptively — shrinking S when reward variance rises and expanding it when the pool is tight — might recover some of the quality lost at large S while retaining most of the cost saving; the paper treats S as fixed.
  • The reported savings depend on the rental-price gap between data-center GPUs and distributed consumer GPUs; if that gap narrows, ECHO-2's cost advantage would shrink accordingly, a sensitivity the paper does not quantify.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. ECHO-2 proposes a distributed RL post-training system that keeps the learner centralized on a small GPU cluster while offloading rollout generation to cheap, heterogeneous, geographically distributed inference workers. The system treats bounded policy staleness S as a user-controlled budget and uses a publication period κ to overlap training, policy dissemination, and rollout generation. The main technical contributions are an overlap-based capacity condition (Eqs. (1)-(2)) for provisioning sufficient aggregate rollout throughput, a peer-assisted pipelined broadcast mechanism, cost-aware worker activation, and a three-plane (Rollout/Learning/Data) abstraction. Experiments on GRPO post-training of Qwen3-4B and Qwen3-8B under simulated WAN bandwidth regimes report a 33.3–36.3% cost reduction at matched AIME accuracy, with reward comparable to centralized verl baselines.

Significance. The cost problem addressed is real and practically important: rollout generation dominates RL post-training cost, and using cheaper distributed inference is a natural way to reduce it. The paper's strengths are its concrete system design, the explicit overlap-based provisioning rule, the peer-assisted broadcast mechanism, cost-aware scheduling, and its honest statement that staleness robustness is only empirical. If the reported cost figures can be reconciled with the paper's own pricing and timing inputs, this would be a useful contribution. However, the headline cost claim is currently not reproducible from the stated numbers, and the §5.4 'validation' of the overlap condition is largely a consistency check rather than an independent prediction. These issues are local and fixable, so the contribution is promising but not yet in publishable form.

major comments (4)
  1. [§5.1, §5.2, Table 2] The cost accounting is internally inconsistent. Using the disclosed formula Cost$ = Σ p_g·GPU-hours_g, Table 1 prices, the stated ECHO-2 hardware (4×A100 learner + 9×RTX 5090 workers), and the reported Ttrain=1631.2s for ECHO-2(S=3), I compute (4×3.06 + 9×0.35)×(1631.2/3600) ≈ $6.97 per training step. Table 2 lists $8.098 for the Full configuration, about 16% higher. If Table 2 refers to a different model, S, or Ttrain, that must be stated; otherwise Fig. 3a's cost axes and the headline 33.3–36.3% savings cannot be verified.
  2. [§3.3, Eq. (1)-(2), §5.4] Eq. (1) is by construction the no-bubble condition: it says the learner consumes κR rollouts in κTtrain and waits at most Tbcast before new-policy rollouts arrive. Eq. (2) is an algebraic rearrangement. Therefore Fig. 3c's observation that the bubble ratio drops near the predicted threshold is a consistency check, not an empirical test of a predictive model. Please reframe the experiment as such, or add an independent prediction (e.g., measured μpool as a function of worker count) that is not guaranteed by the definition of no-bubble.
  3. [Appendix A.3, Eq. (11)-(12)] The derivation contains an invalid ceiling/floor step. From (Tbcast+R/μpool)/Ttrain ≤ 1 + (1−1/κ)Tbcast/Ttrain, it does not follow that ceil(...) ≤ 1 + floor((1−1/κ)Tbcast/Ttrain); for example, with X=0.6 one gets ceil(1.6)=2 > 1+floor(0.6)=1. The bound should use ceilings (ceil(1+X) ≤ 1+ceil(X)), and the claimed Δ≤3 for κ=2 must be re-derived. The final bound may still be true, but the proof as written is incorrect.
  4. [§5.2, Table 2] The main cost comparison changes hardware and distribution simultaneously: ECHO-2 uses 4×A100 + 9×RTX 5090, while Centralized-Sync/Async use 8×A100. Some of the reported saving is likely due to substituting cheap 5090s for expensive A100s in the rollout plane. The internal ablations show that P2P broadcast and cost-aware activation reduce cost within ECHO-2, but there is no co-located cheap-rollout baseline (e.g., 4×A100 + 9×5090 without WAN/P2P dissemination) to quantify how much of the saving is attributable to ECHO-2's mechanisms. Adding such a baseline or explicitly decomposing the saving into hardware substitution versus system overhead would strengthen the central claim.
minor comments (6)
  1. [Abstract] The abstract in the submission header says 'LLMs ranging from 4B to 32B parameters', while the paper body and the abstract in the full text say 4B and 8B models. Align these statements.
  2. [§5.1, §5.2] Network settings are inconsistent: §5.1 says B0 ∈ {unlimited, 300–1000 Mbps} and Bw=100 Mbps, but §5.2 says B0=100 Mbps and Bw=1 Gbps. Clarify which caps were actually used for the cost–quality experiments.
  3. [Table 2] Table 2 does not state the model, the staleness budget S, or Ttrain per row. The caption should specify how 'Cost/Step' is computed, including whether it includes only steady-state Ttrain or also wait time and dissemination time.
  4. [§3.2, §4.1] The statement that a policy version is updated 'after each training step, which includes two model updates' is unclear. Define whether 'training step' and 'model update' are distinct units, and how versions increment relative to the two updates.
  5. [§5.5.1] The target fraction is written as q=1/γ=1/1.1=0.9. Since γ is a safety factor on capacity, its reciprocal has no obvious relation to a dissemination target fraction; define q independently.
  6. [§5.1, Eq. (2)] The per-worker effective throughput μ_i and the aggregate μpool used in the experiments are not reported. Providing these values would make the capacity rule reproducible and help readers interpret the threshold in Fig. 3c.

Circularity Check

1 steps flagged · score 6.0 of 10

Capacity rule is a rearranged no-bubble definition, so the §5.4 threshold 'prediction' is a consistency check; the Table 2 cost mismatch is a reproducibility issue, not circularity.

  1. self definitional [§3.3 Eq. (1)–(2); §5.4 Fig. 3c]
    "To avoid training bubbles, rollout generation and dissemination must be completed within one publication period: κTtrain ≥ Tbcast + κR / Σμi, (1) ... Rearranging yields an aggregate capacity requirement: Σμi ≥ μmin(κ) ≜ κR / (κTtrain − Tbcast), κTtrain > Tbcast. (2) ... equation (2) predicts a threshold behavior: as rollout capacity increases, learner bubbles should rapidly vanish once the system enters the feasible overlap region."

    Equation (2) is the algebraic rearrangement of equation (1), and equation (1) is precisely the definition of a bubble-free publication period: 'to avoid training bubbles' means the dissemination time plus the time to generate κR rollouts must fit inside κTtrain. The §5.4 claim that equation (2) 'predicts' bubble ratios collapsing at the threshold is therefore not an independent empirical prediction; it is the same no-bubble condition restated. The observed bubble ratios can confirm that Ttrain, Tbcast, and μi were measured consistently, but the threshold location is fixed by construction, not discovered from the bubble data.

full rationale

ECHO-2's central cost-efficiency and RL-quality claims rest on a real empirical comparison against centralized verl baselines, with independently reported AIME/benchmark scores and cost curves; those comparisons are not circular. The overlap/capacity model, however, is presented as predictive in §5.4 but is a direct rearrangement of the definition of a training bubble, so its 'validation' is a consistency check rather than a test of a separate prediction. The staleness corollary in Appendix A is a legitimate derivation from stated worst-case assumptions and the overlap condition; it does not smuggle in the target result. Citations to Parallax [14], ECHO [11], and prior asynchronous RL work [7,8,10,15,21] are used as implementation or motivation, not as an unverified uniqueness theorem, so no self-citation load-bearing circularity is present. The Table 2 cost/step value is not reproducible from the paper's own Cost$ formula and stated prices (about $6.97/step for the S=3 Full configuration versus the reported $8.098), but that is a correctness/reproducibility discrepancy rather than a definitional reduction. Overall, one explicit 'prediction' reduces by construction, so a score of 6 is appropriate; the independent experimental content prevents a higher score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced. The framework itself relies on measured system quantities (Ttrain, Tbcast, μ_i) and user-chosen control parameters (S, κ, γ). The most serious load-bearing premise is the empirical staleness robustness of GRPO, which the authors themselves do not guarantee, plus a pricing/accounting inconsistency that affects the headline cost numbers.

free parameters (3)
  • staleness budget S = swept over 3,4,6,11 (8B) and 3,6,9,11,21 (4B)
    User-specified control parameter; the safe range is asserted empirically, and the paper states no formal guarantees (§6). It directly controls the cost-quality trade-off and the default publication period κ=S−1.
  • publication period κ = S−1 by default
    Derived from S by a conservative bound, but still a design parameter that trades snapshot freshness against dissemination overhead.
  • safety factor γ = 1.1
    Chosen by hand in §4.3 to inflate the target capacity μ_target = γμ_min and absorb variability; affects when workers are activated or released.
assumptions (5)
  • domain assumption Modern LLM RL objectives tolerate bounded policy staleness without degrading final quality.
    Load-bearing; invoked in §3.1 and the whole design, but explicitly flagged in §6 as having no formal guarantee and being task/reward dependent.
  • domain assumption Tbcast/Ttrain < 1 in all experimental settings.
    Stated in the Appendix A corollary and §3.3; needed for κ=S−1 to imply Δmax≤S. No per-experiment values of Tbcast/Ttrain are reported.
  • domain assumption A worker's effective throughput μ_i captures inference, reward computation, scheduling, network latency, and straggler effects end-to-end.
    Stated in §3.2. If μ_i is optimistic, the capacity rule under-provisions and bubbles appear.
  • domain assumption The only sources of learner idle time are insufficient admissible rollouts and dissemination latency.
    The overlap condition Eq. (1) models no other bubble sources; real systems also have data-plane and scheduling overheads.
  • domain assumption Rental prices in Table 1 (A100 $3.06/h, RTX5090 $0.35/h) accurately represent the costs used in the headline savings.
    The central cost-efficiency claim is computed from these prices, but the cost per step in Table 2 does not reconcile with them.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning." pith.science (2026). https://pith.science/paper/FQB5Z6HR

@misc{pith2026260202192,
  author       = {Pith},
  title        = {Pith review of: ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FQB5Z6HR}},
  note         = {Machine review of arXiv:2602.02192}
}
read the original abstract

Reinforcement learning (RL) is a critical stage in post-training large language models (LLMs), involving repeated interaction between rollout generation, reward evaluation, and centralized learning. Distributing rollout execution offers opportunities to leverage more cost-efficient inference resources, but introduces challenges in wide-area coordination and policy dissemination. We present ECHO-2, a distributed RL framework for post-training with remote inference workers and non-negligible dissemination latency. ECHO-2 combines centralized learning with distributed rollouts and treats bounded policy staleness as a user-controlled parameter, enabling rollout generation, dissemination, and training to overlap. We introduce an overlap-based capacity model that relates training time, dissemination latency, and rollout throughput, yielding a practical provisioning rule for sustaining learner utilization. To mitigate dissemination bottlenecks and lower cost, ECHO-2 employs peer-assisted pipelined broadcast and cost-aware activation of heterogeneous workers. Experiments on GRPO post-training of LLMs ranging from 4B to 32B parameters under real wide-area bandwidth regimes show that ECHO-2 significantly improves cost efficiency while preserving RL reward comparable to strong baselines.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Training

    cs.AI 2026-06 conditional novelty 6.0 of 10

    Dynamically reassigning GPUs between rollout and training during LLM post-training cuts pipeline bubbles, improving measured throughput by 66.5% over the best static split in the paper's 8-GPU test.

Reference graph

Works this paper leans on

32 extracted references · 12 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    arXiv 2025

  2. [2]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  3. [3]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    arXiv 2024

  4. [4]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017

    2017

  5. [5]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

    2024

  6. [6]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. InProceedings of the TwentiethEuropean Conference on Computer Systems, pages 1279–1297, 2025

    2025

  7. [7]

    Areal: A large-scale asynchronous reinforcement learning system for language reasoning.arXiv preprint arXiv:2505.24298, 2025

    Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning.arXiv preprint arXiv:2505.24298, 2025

    arXiv 2025

  8. [8]

    History rhymes: Accelerating llm reinforcement learning with rhymerl.arXiv preprint arXiv:2508.18588, 2025

    Jingkai He, Tianjian Li, Erhu Feng, Dong Du, Qian Liu, Tao Liu, Yubin Xia, and Haibo Chen. History rhymes: Accelerating llm reinforcement learning with rhymerl.arXiv preprint arXiv:2508.18588, 2025

    arXiv 2025

Show all 32 references
  1. [9]

    Asynchronous rlhf: Faster and more efficient off-policy rl for language models.arXiv preprint arXiv:2410.18252, 2024

    Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous rlhf: Faster and more efficient off-policy rl for language models.arXiv preprint arXiv:2410.18252, 2024

    2024 arXiv

  2. [10]

    Streamrl: Scalable, heterogeneous, and elastic rl for llms with disaggregated stream generation

    Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, et al. Streamrl: Scalable, heterogeneous, and elastic rl for llms with disaggregated stream generation. arXiv preprint arXiv:2504.15930, 2025

    2025 arXiv

  3. [11]

    Echo: Decoupling inference and training for large-scale rl alignment on heterogeneous swarms

    Jie Xiao, Changyuan Fan, Qingnan Ren, Alfred Long, Yuchen Zhang, Rymon Yu, Eric Yang, Lynn Ai, and Shaoduo Gan. Echo: Decoupling inference and training for large-scale rl alignment on heterogeneous swarms. arXiv preprint arXiv:2508.05387, 2025

    2025 arXiv

  4. [12]

    Petals: Collaborative inference and fine-tuning of large models

    Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Maksim Riabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. Petals: Collaborative inference and fine-tuning of large models. InProceedings of the 61st Annual Meeting of the Association for Computation...

    2023

  5. [13]

    Swarm parallelism: Training large models can be surprisingly communication-efficient

    Max Ryabinin, Tim Dettmers, Michael Diskin, and Alexander Borzunov. Swarm parallelism: Training large models can be surprisingly communication-efficient. InInternational Conference on Machine Learning, pages 29416–29440. PMLR, 2023

    2023

  6. [14]

    Parallax: Efficient llm inference service over decentralized environment.arXiv preprint arXiv:2509.26182, 2025

    Chris Tong, Youhe Jiang, Gufeng Chen, Tianyi Zhao, Sibian Lu, Wenjie Qu, Eric Yang, Lynn Ai, and Binhang Yuan. Parallax: Efficient llm inference service over decentralized environment.arXiv preprint arXiv:2509.26182, 2025

    2025

  7. [15]

    Rlax: Large-scale, distributed reinforcement learning for large language models on tpus.arXiv preprint arXiv:2512.06392, 2025

    Runlong Zhou, Lefan Zhang, Shang-Chen Wu, Kelvin Zou, Hanzhi Zhou, Ke Ye, Yihao Feng, Dong Yin, Alex Guillen Garcia, Dmytro Babych, et al. Rlax: Large-scale, distributed reinforcement learning for large language models on tpus.arXiv preprint arXiv:2512.06392, 2025

    2025

  8. [16]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  9. [17]

    A survey of reinforcement learning from human feedback, 2024

    Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback, 2024

    2024

  10. [18]

    Areal-hex: Accommodating asynchronous rl training over heterogeneous gpus.arXiv preprint arXiv:2511.00796, 2025

    Ran Yan, Youhe Jiang, Tianyuan Wu, Jiaxuan Gao, Zhiyu Mei, Wei Fu, Haohui Mai, Wei Wang, Yi Wu, and Binhang Yuan. Areal-hex: Accommodating asynchronous rl training over heterogeneous gpus.arXiv preprint arXiv:2511.00796, 2025

    2025

  11. [19]

    Intellect-2: A reasoning model trained through globally decentralized reinforcement learning.arXiv preprint arXiv:2505.07291, 2025

    Prime Intellect Team, Sami Jaghouar, Justus Mattern, Jack Min Ong, Jannik Straube, Manveer Basra, Aaron Pazdera, Kushal Thaman, Matthew Di Ferrante, Felix Gabriel, et al. Intellect-2: A reasoning model trained through globally decentralized reinforcement learning.arXiv preprin...

    2025 arXiv

  12. [20]

    Qiu, and Yuqing Yang

    Xufang Luo, Yuge Zhang, Zhiyuan He, Zilong Wang, Siyun Zhao, Dongsheng Li, Luna K. Qiu, and Yuqing Yang. Agent lightning: Train any ai agents with reinforcement learning, 2025

    2025

  13. [21]

    Prosperity before collapse: How far can off-policy rl reach with stale data on llms?arXiv preprint arXiv:2510.01161, 2025

    Haizhong Zheng, Jiawei Zhao, and Beidi Chen. Prosperity before collapse: How far can off-policy rl reach with stale data on llms?arXiv preprint arXiv:2510.01161, 2025

    2025

  14. [22]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  15. [23]

    American invitational mathematics examination (AIME), 2024

    MAA. American invitational mathematics examination (AIME), 2024

    2024

  16. [24]

    Omni-math: A universal olympiad level mathematic benchmark for large language models

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985, 2024

    2024 arXiv

  17. [25]

    Have llms advanced enough? a challenging problem solving benchmark for large language models

    Daman Arora, Himanshu Singh, et al. Have llms advanced enough? a challenging problem solving benchmark for large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7527–7543, 2023

    2023

  18. [26]

    Hardmath: A benchmark dataset for challenging problems in applied mathematics

    Jingxuan Fan, Sarah Martinson, Erik Y Wang, Kaylie Hausknecht, Jonah Brenner, Danxian Liu, Nianli Peng, Corey Wang, and Michael P Brenner. Hardmath: A benchmark dataset for challenging problems in applied mathematics. arXiv preprint arXiv:2410.09988, 2024

    2024 arXiv

  19. [27]

    Towards robust mathematical reasoning

    Minh-Thang Luong, Dawsen Hwang, Hoang H Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, et al. Towards robust mathematical reasoning. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Proce...

    2025

  20. [28]

    Qwen3 technical report, 2025

    Qwen Team. Qwen3 technical report, 2025

    2025

  21. [29]

    Openai gpt-5 system card, 2025

    OpenAI. Openai gpt-5 system card, 2025

    2025

  22. [30]

    Grok 4 model card

    xAI. Grok 4 model card. Technical report, xAI, 2025

    2025

  23. [31]

    Claude sonnet 4.5 system card

    Anthropic. Claude sonnet 4.5 system card. Technical report, Anthropic, 2025

    2025

  24. [32]

    moba://hok_v2

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. 14 A Worst-Case Staleness Bound under Overlap This appendix derives a conservative u...

    2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.