Incentivizing Permissionless Distributed Learning of LLMs

Amir Sarfi; Eugene Belilovsky; Evangelos Pappas; Jacob Steeves; Joel Lidin; Samuel Dare

REVIEW 3 major objections 4 minor 1 cited by

Incentivizing Permissionless Distributed Learning of LLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces Gauntlet, a token-based incentive system that pays anonymous, unvetted peers for pseudo-gradient updates to a shared model, and reports that it trained a 1.2B-parameter LLM in what the authors call the first fully…

desk verdict A genuinely deployed permissionless training incentive system with real token payouts, but the scoring equations are inconsistent and the empirical support is thinner than the claims. read the letter →

arxiv 2505.21684 v1 pith:LPIH7UVW submitted 2025-05-27 cs.LG cs.DC

Joel Lidin , Amir Sarfi , Evangelos Pappas , Samuel Dare , Eugene Belilovsky , Jacob Steeves This is my paper · ORCID

classification cs.LGcs.DC

keywords permissionlessdistributedtrainingincentivemechanismpseudo-gradientaggregationloss-basedscoringOpenSkillratingproofofcomputationcommunication-efficientoptimizationdecentralizedLLMpre-training

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Gauntlet is an incentive mechanism for distributed deep learning in which anyone can contribute compressed update vectors, called pseudo-gradients, to a shared model, without registration or vetting. The paper's central claim is that Gauntlet, paired with the communication-efficient DeMo optimizer, encourages high-quality contributions from anonymous peers around the world and produced a 1.2B-parameter language model that converges on a per-iteration basis against a centralized AdamW baseline. The core idea is to estimate, for each contributing peer, how much its pseudo-gradient reduces the model's loss on a random data subset, convert those noisy measurements into an OpenSkill rating, and combine the rating with a proof-of-computation signal that checks whether the peer trained on its assigned data. Tokens are paid proportionally to the resulting score, and only top-scoring peers are aggregated into the model update. A live 20,000-round run paid real token rewards to participants, which the authors describe as the first truly permissionless pre-training LLM run.

What carries the argument

The load-bearing object is the LossScore $\mathrm{LossScore}_p(\Delta^p_t, D) = L(\theta_t, D) - L(\theta_t - \beta\Delta^p_t, D)$, which estimates how much a peer's update improves the loss on a random subset. Because this score is noisy, it feeds an OpenSkill rating system, a pairwise ranking algorithm, rather than being used directly as money. A second mechanism, the proof-of-computation signal $\mu_p$, is an exponential moving average of the sign of the difference between the peer's loss improvement on its assigned data and on random data; it detects whether the peer actually trained on its assigned data. Fast evaluation filters peers on uptime, formatting, and a synchronization score computed from a tiny sample of model parameters. The product $\mu_p \cdot \mathrm{LossRating}_p$ is normalized and top-$G$ peers are aggregated with equal weight.

What would settle it

Run the protocol in a controlled environment where the true quality of each pseudo-gradient is known, for example one peer trained on extra data, one on stale data, and one on random noise, and check whether the peer score ranks them in that order. If the ranking repeatedly disagrees with ground truth or is no better than random subsets of data, the loss-score pipeline is rewarding evaluator noise rather than useful computation.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that loss-based valuation can turn anonymous, unvetted contributors into a viable training collective. The mechanism computes, for each sampled peer, the loss drop produced by its pseudo-gradient $\Delta^p_t$ on a random data subset $D$ (Eq. (2)), feeds sparse noisy scores into an OpenSkill rating so rankings stay meaningful over time, and multiplies that rating by a proof-of-computation signal $\mu_p$ that compares loss on data assigned to the peer with loss on random data. The product is the peer score that determines token payouts and the top-$G$ set whose updates get aggregated. The paper reports that a live 20,000-round permissionless run of a 1.2B model converged comparably, per iteration, to a centralized AdamW baseline, with real token rewards paid out.

Load-bearing premise

The whole incentive pipeline assumes that the loss difference measured on a random data subset reliably ranks peers by the true quality of their pseudo-gradients, even though the paper shows that this score is noisy and inconsistent across adjacent rounds.

Editorial extensions

If this is right

Gauntlet can be attached to any synchronous scheme that aggregates updates or pseudo-gradients, not just DeMo, because it only needs a validator to apply each contribution to the current model and measure the loss.
Peers are incentivized to process more data and keep their local model synchronized, because both behaviors directly improve LossRating and the proof-of-computation signal.
Because incentives come from loss improvement rather than from matching a prescribed computation, peers are free to innovate on data selection or local hyperparameters; the mechanism rewards useful deviation instead of punishing it.
The sign and normalization in aggregation limit the damage from byzantine peers, although the paper concedes that a single oversized malicious update sent before downweighting can still disrupt a round.
Deployment on a blockchain with staked validators keeps the incentive record public, so token payouts can be audited after the fact.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implication the paper leaves implicit is that the validator's choice of the random subset $D$ is the control point of the whole system; whoever chooses evaluation data effectively chooses which peers earn tokens, so a natural hardening would be a commit-reveal protocol where the validator commits to $D$ before peer submissions arrive.
If the loss-score ranking transfers across data heterogeneity, the same evaluation could reward other useful contributions, such as curated data, better local schedules, or adapted optimizers, turning training into an open market for any update that lowers loss.
The per-iteration comparison to AdamW is not a per-token comparison because token counts per peer are unknown; an independent measurement of actual compute and tokens would show whether Gauntlet selected genuinely better updates or simply rewarded scale and luck.
A simple offline test would settle the central mechanism: replay logged contributions from the live run and compare the loss improvement of the Gauntlet-selected top-$G$ set against $G$ randomly chosen peers, holding the number of contributions fixed.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Candid take: the paper reports something real—a live permissionless training run of a 1.2B LLM on Bittensor with token incentives—and the integration of loss-based scoring, OpenSkill, assigned-data proof-of-computation, and sync filtering is new as a package. That deserves credit. But the central scoring equation is inconsistent between the math and the pseudocode, and the empirical claims are thinner than the prose.

What's new and good: the deployment itself is the main contribution. Gauntlet has been running with real peers and real token payouts, and the model converges. The assigned-data trick to prevent copying is sensible, and the fast-sync filter is pragmatic. The authors are honest that token counts are estimated and that peer compute is uncontrolled. The small simulation does show the intended behavior: more data gets higher rating, desync gets downweighted.

Soft spots: the biggest is the mismatch between Eq. (2) and Algorithm 1. Eq. (2) defines LossScore using a beta-scaled step, beta = c*alpha with c<1, explicitly to avoid noisy rankings. Algorithm 1 evaluates theta' = theta_t - alpha*sign(Delta_p), a full learning-rate signed step. Sign normalization discards magnitude, and an alpha-sized step is exactly what the text says leads to inconsistent rankings. The paper never says which variant was deployed. That matters: if they used Algorithm 1, the score may reward luck; if they used Eq. (2), the evaluation doesn't match the signed aggregation used for training. Either way, the claim that incentives selected high-quality contributions isn't established by the evidence presented.

The empirical support is otherwise thin. The AdamW baseline is not matched on tokens or compute, the benchmark table has no error bars, and the only simulation is 400 steps with no seeds. The "competitive" claim is per iteration, not final model quality. These are fixable with more careful experiments and reporting, not fatal to the idea.

Summary: this paper is for people working on decentralized training and cryptoeconomic incentives. It's a useful existence proof that permissionless training can be run and can converge. It deserves a serious referee, but needs major revision: clarify the scoring implementation, report which variant was deployed, add variance and proper baselines, and ideally release logs and code so the loss-score reliability can be checked.

Recommendation: send it to review, but expect heavy revision. If the authors nail down the scoring consistency and show variance of the loss-score estimator, it could become a useful reference.

Referee Report

3 major / 4 minor

Summary. The paper proposes Gauntlet, an incentive mechanism for synchronous permissionless distributed training of LLMs. Peers contribute compressed pseudo-gradients, and a validator scores each contribution by its effect on the loss, maintains an OpenSkill rating over time, and combines this rating with a proof-of-computation signal to issue token rewards and select the top-G peers for aggregation. The authors report a live 1.2B-parameter run on the Bittensor blockchain with permissionless peers, compare its training curve and zero-shot benchmarks against AdamW and DeMo baselines, and claim this is the first truly permissionless pre-training LLM run.

Significance. If the central claims hold, the paper provides a valuable real-world demonstration that anonymous, unvetted participants can collectively train a competitive model under token-based incentives. The deployment over 20K communication rounds with real payouts is a distinctive empirical contribution, and the high-level mechanism is clearly and compactly described. The main significance is therefore contingent on resolving the ambiguity between the theoretical loss score and the deployed evaluation procedure, and on providing stronger evidence that the loss-based ranking reliably identifies useful pseudo-gradients. The paper is transparent about several uncertainties, such as uncontrolled participant compute and estimated token counts, which is commendable, but those uncertainties currently limit the strength of the comparative claims.

major comments (3)

[Section 3.1 and Algorithm 1] Equation (2) defines LossScore with the scaled step βΔ_p, and the text states that β = c·α with c < 1 because 'stepping with too large a step size is more likely to lead to negative loss scores, and in our empirical observations inconsistent rankings between peers.' Algorithm 1, however, evaluates the candidate with θ'_p = θ_t − α·SIGN(Δ_p) and uses this full signed step in the definitions of δ_assigned and δ_rand. A signed step discards gradient magnitude and is not the β-scaled update of Eq. (2), so the two definitions are not equivalent. The manuscript does not state which evaluation was used in the live 1.2B run, nor does it report the variance or rank correlation of the resulting scores. This is load-bearing because LossRating, µ, top-G selection, and token payouts all flow from this score; please reconcile the pseudocode with Eq. (2) and provide score-noise statistics for the deployed variant.
[Section 6, Figure 2] The controlled simulation supporting the claim that LossRating 'can quickly differentiate between peers' uses three peers and 400 steps, with no error bars, no seeds, and no measure of ranking stability. Given that the paper itself states in Section 3.1 that loss scores are not consistent over time and can vary across adjacent iterates, this simulation is too weak to establish that OpenSkill-based ranking reliably distinguishes high-quality contributions under realistic noise. Please report multi-seed results, the distribution of LossScore for identical peers, and a rank-correlation or top-G stability metric.
[Table 1 and Figure 1] The comparison of TEMPLAR-1B to the AdamW baseline is not matched on tokens or compute: TEMPLAR-1B tokens are estimated at 100B-200B, the number of tokens actually processed by participants is uncontrolled, and Table 1 reports no error bars, checkpoint selection, or evaluation-harness details. The claim of 'competitive (on a per-iteration basis)' may be true, but as presented the reader cannot distinguish the effect of the incentive mechanism from additional compute or data. Please report fixed-compute or fixed-token matched baselines, or explicitly state this limitation and soften the comparative wording.

minor comments (4)

[Algorithm 1] The Require line lists 'EMA decay β', but Eq. (3) and the pseudocode use γ for the EMA decay; please use a single consistent symbol to avoid confusion with the β step-scaling factor in Eq. (2).
[Table 1] The caption acknowledges that token counts are estimated, but the table would be easier to interpret if it also stated whether evaluations are zero-shot with a standard harness (e.g., lm-eval) and which checkpoint was used.
[Figure 2] The left panel is labeled 'Gradient Score' while the text refers to LossScore; please align the axis label with Eq. (2).
[Section 1 and Section 7] The claim that this is 'the first truly permissionless pre-training LLM run' is a strong historical assertion; it would benefit from a precise definition of 'permissionless' and a discussion of why prior decentralized efforts such as [3], [6], and [16] do not qualify.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the incentive mechanism is validated by external benchmarks; only a minor, non-load-bearing self-citation to Bittensor/Yuma consensus appears.

full rationale

The central claim—that Gauntlet's loss-based LossScore (Eq. 2) rewards useful pseudo-gradients and that this produces a competitive permissionless 1.2B run—does not reduce to its own inputs. The final model is judged on zero-shot benchmarks (HellaSwag, PIQA, ARC-E) that are external to the scoring signal, so the demonstrated convergence is not equivalent to the reward function by construction. No fitted constant is relabeled as a prediction: beta = c*alpha is set as a heuristic with c<1 and, while not thoroughly validated, it is not fit to the reported evaluation metrics. The paper imports no uniqueness theorem and does not smuggle in its ansatz via self-citation; the sign operation is attributed to the external DeMo work [12]. The only self-citation is [18], by co-author Steeves, used as background for Yuma consensus, with the paper explicitly stating that a full description is beyond scope; this is not load-bearing for any derivation. A real concern is the textual inconsistency between Eq. (2) (step beta*Delta_p with beta < alpha) and Algorithm 1 (step alpha*SIGN(Delta_p)), which could threaten the validity of the deployed scoring, but this is an internal-consistency/robustness issue, not a circularity in the sense of a conclusion equivalent to its premises.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The mechanism rests on several untested domain assumptions about loss-score reliability, i.i.d. data, and validator honesty, plus hand-chosen hyperparameters. The central empirical claim depends on these assumptions rather than on a fitted prediction, so circularity is low but robustness is not proven.

free parameters (6)

Beta scaling factor for LossScore = beta_t = c * alpha_t with c < 1 (c unspecified)
Controls the step size in Eq. 2; chosen empirically to reduce noise and ranking inconsistency, not derived.
EMA decay gamma for mu_p = unspecified
In Eq. 3, controls how fast the proof-of-computation signal accumulates; no value or sensitivity analysis is given.
Fast-evaluation penalty phi = 0.75
Multiplicative penalty applied to mu_p when a peer fails a fast evaluation check; hand-chosen.
Sync score threshold = 3
Threshold used as a filter in fast evaluation; chosen in practice with no analysis.
Normalization exponent c = 2
In Eq. 5, increases competition among peers; hand-chosen.
Aggregation top-G group size = G = 15
Number of top peers aggregated each round; engineering choice. Primary evaluation set size is v = 5 peers per round.

assumptions (6)

domain assumption Loss computed on a random subset D of the training set is a faithful, low-variance estimator of a pseudo-gradient's contribution quality.
Used in Eq. 2 and OpenSkill ranking; no variance analysis is provided, and the paper itself notes loss scores are highly variable across adjacent iterates.
domain assumption All valid peers train on i.i.d. data subsets with comparable pseudo-gradient norms.
Assumed in Algorithm 2 line 12 to justify normalization; the authors state they do not anticipate large variations in the norms of valid pseudo-gradients.
domain assumption Peers synchronously maintain the validator's model state, so loss-score comparisons are across comparable models.
Relies on the sync score and coordinated aggregation; the paper says synchronized states are critical for comparing contributions.
domain assumption Bittensor's Yuma consensus and stake mechanism ensure validators are honest and provide a reliable global clock.
Invoked in Section 3.3 and cited to reference [18]; not independently verified in this paper.
domain assumption OpenSkill ratings convert sparse, noisy pairwise loss-score comparisons into a reliable ranking.
Used for LossRating; the authors cite OpenSkill but do not validate its behavior on this specific sparse-evaluation distribution.
domain assumption Training on assigned data Dp_t yields a detectable positive sign of LossScore(Dp_t) minus LossScore(Drand_t) over time.
Foundation of the proof-of-computation signal in Eq. 3; the only support is a small simulation, not a theoretical guarantee.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Incentivizing Permissionless Distributed Learning of LLMs." pith.science (2026). https://pith.science/paper/LPIH7UVW

@misc{pith2026250521684,
  author       = {Pith},
  title        = {Pith review of: Incentivizing Permissionless Distributed Learning of LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LPIH7UVW}},
  note         = {Machine review of arXiv:2505.21684}
}

read the original abstract

We describe an incentive system for distributed deep learning of foundational models where peers are rewarded for contributions. The incentive system, \textit{Gauntlet}, has been deployed on the bittensor blockchain and used to train a 1.2B LLM with completely permissionless contributions of pseudo-gradients: no control over the users that can register or their hardware. \textit{Gauntlet} can be applied to any synchronous distributed training scheme that relies on aggregating updates or pseudo-gradients. We rely on a two-stage mechanism for fast filtering of peer uptime, reliability, and synchronization, combined with the core component that estimates the loss before and after individual pseudo-gradient contributions. We utilized an OpenSkill rating system to track competitiveness of pseudo-gradient scores across time. Finally, we introduce a novel mechanism to ensure peers on the network perform unique computations. Our live 1.2B run, which has paid out real-valued tokens to participants based on the value of their contributions, yielded a competitive (on a per-iteration basis) 1.2B model that demonstrates the utility of our incentive system.

Figures

Figures reproduced from arXiv: 2505.21684 by the authors.

**Figure 2.** Simulating how LossScore and LossRating evolve for three peers: one processing more [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Overcoming the Communication-Performance Tradeoff in LLM Pretraining
cs.LG 2025-08 conditional novelty 7.0 of 10

SparseLoCo combines error feedback with Top-k sparsification and 2-bit quantization to send 1-3% of the pseudo-gradient during LLM pre-training while matching or beating DiLoCo's dense updates.

Reference graph

Works this paper leans on

20 extracted references · 9 canonical work pages · cited by 1 Pith paper

[1]

Dion: A communication-efficient optimizer for large models

Kwangjun Ahn and Byron Xu. Dion: A communication-efficient optimizer for large models. arXiv preprint arXiv:2504.05295, 2025

arXiv 2025
[2]

Verde: Verification via refereed delegation for machine learning programs

Arasu Arun, Adam St Arnaud, Alexey Titov, Brian Wilcox, Viktor Kolobaric, Marc Brinkmann, Oguzhan Ersoy, Ben Fielding, and Joseph Bonneau. Verde: Verification via refereed delegation for machine learning programs. arXiv preprint arXiv:2502.19405, 2025

arXiv 2025
[3]

Training transformers together

Alexander Borzunov, Max Ryabinin, Tim Dettmers, Quentin Lhoest, Lucile Saulnier, Michael Diskin, Yacine Jernite, and Thomas Wolf. Training transformers together. In NeurIPS 2021 Competitions and Demonstrations Track, pages 335–342. PMLR, 2022

work page 2021
[4]

Diloco: Distributed low- communication training of language models

Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low- communication training of language models. arXiv preprint arXiv:2311.08105, 2023

arXiv 2023
[5]

Accurate, large minibatch sgd: Training imagenet in 1 hour

Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017

arXiv 2017
[6]

Intellect-1 technical report

Sami Jaghouar, Jack Min Ong, Manveer Basra, Fares Obeid, Jannik Straube, Michael Keiblinger, Elie Bakouch, Lucas Atkins, Maziyar Panahi, Charles Goddard, et al. Intellect-1 technical report. arXiv preprint arXiv:2412.01152, 2024

arXiv 2024
[7]

Proof-of-learning: Definitions and practice

Hengrui Jia, Mohammad Yaghini, Christopher A Choquette-Choo, Natalie Dullerud, Anvith Thudi, Varun Chandrasekaran, and Nicolas Papernot. Proof-of-learning: Definitions and practice. In 2021 IEEE Symposium on Security and Privacy (SP) , pages 1039–1056. IEEE, 2021

work page 2021
[8]

OpenSkill: A faster asymmetric multi-team, multiplayer rating system

Vivek Joshy. Openskill: A faster asymmetric multi-team, multiplayer rating system. arXiv preprint arXiv:2401.05451, 2024

work page Pith review arXiv 2024

Show all 20 references

[9]

Error feedback fixes signsgd and other gradient compression schemes

Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In International Conference on Machine Learning, pages 3252–3261. PMLR, 2019

2019
[10]

Byzantine robustness and partial participation can be achieved at once: Just clip gradient differences.arXiv preprint arXiv:2311.14127, 2023

Grigory Malinovsky, Peter Richtárik, Samuel Horváth, and Eduard Gorbunov. Byzantine robustness and partial participation can be achieved at once: Just clip gradient differences.arXiv preprint arXiv:2311.14127, 2023

2023 arXiv
[11]

The fineweb datasets: Decanting the web for the finest text data at scale

Guilherme Penedo, Hynek Kydlí ˇcek, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro V on Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems , 37:30811–30849, 2024

2024
[12]

Decoupled momentum optimization

Bowen Peng, Jeffrey Quesnelle, and Diederik P Kingma. Decoupled momentum optimization. arXiv preprint arXiv:2411.19870, 2024

2024
[13]

Robust aggregation for federated learning

Krishna Pillutla, Sham M Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. IEEE Transactions on Signal Processing, 70:1142–1154, 2022

2022
[14]

Error compensated distributed sgd can be acceler- ated

Xun Qian, Peter Richtárik, and Tong Zhang. Error compensated distributed sgd can be acceler- ated. Advances in Neural Information Processing Systems , 34:30401–30413, 2021

2021
[15]

Adaptive federated optimization

Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koneˇcn`y, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020

2003 arXiv
[16]

The future of large language model pre-training is federated

Lorenzo Sani, Alex Iacob, Zeyu Cao, Bill Marino, Yan Gao, Tomas Paulik, Wanru Zhao, William F Shen, Preslav Aleksandrov, Xinchi Qiu, et al. The future of large language model pre-training is federated. arXiv preprint arXiv:2405.10853, 2024

2024 arXiv
[17]

Understanding top-k sparsifica- tion in distributed deep learning

Shaohuai Shi, Xiaowen Chu, Ka Chun Cheung, and Simon See. Understanding top-k sparsifica- tion in distributed deep learning. arXiv preprint arXiv:1911.08772, 2019. 10

1911 arXiv
[18]

Incentivizing intelligence: The bittensor approach, 2022

Jacob Steeves, Ala Shaabana, Yuqian Hu, Francois Luus, Sin Tai Liu, and Jacqueline Dawn Tasker-Steeves. Incentivizing intelligence: The bittensor approach, 2022

2022
[19]

Cocktailsgd: Fine-tuning foundation models over 500mbps networks

Jue Wang, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang, Christopher De Sa, Christopher Re, and Ce Zhang. Cocktailsgd: Fine-tuning foundation models over 500mbps networks. In International Conference on Machine Learning , pages 36058–36076. PMLR, 2023

2023
[20]

Generalized byzantine-tolerant sgd

Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. Generalized byzantine-tolerant sgd. arXiv preprint arXiv:1802.10116, 2018. 11

2018 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Dion: A communication-efficient optimizer for large models

Kwangjun Ahn and Byron Xu. Dion: A communication-efficient optimizer for large models. arXiv preprint arXiv:2504.05295, 2025

arXiv 2025

[2] [2]

Verde: Verification via refereed delegation for machine learning programs

Arasu Arun, Adam St Arnaud, Alexey Titov, Brian Wilcox, Viktor Kolobaric, Marc Brinkmann, Oguzhan Ersoy, Ben Fielding, and Joseph Bonneau. Verde: Verification via refereed delegation for machine learning programs. arXiv preprint arXiv:2502.19405, 2025

arXiv 2025

[3] [3]

Training transformers together

Alexander Borzunov, Max Ryabinin, Tim Dettmers, Quentin Lhoest, Lucile Saulnier, Michael Diskin, Yacine Jernite, and Thomas Wolf. Training transformers together. In NeurIPS 2021 Competitions and Demonstrations Track, pages 335–342. PMLR, 2022

work page 2021

[4] [4]

Diloco: Distributed low- communication training of language models

Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low- communication training of language models. arXiv preprint arXiv:2311.08105, 2023

arXiv 2023

[5] [5]

Accurate, large minibatch sgd: Training imagenet in 1 hour

Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017

arXiv 2017

[6] [6]

Intellect-1 technical report

Sami Jaghouar, Jack Min Ong, Manveer Basra, Fares Obeid, Jannik Straube, Michael Keiblinger, Elie Bakouch, Lucas Atkins, Maziyar Panahi, Charles Goddard, et al. Intellect-1 technical report. arXiv preprint arXiv:2412.01152, 2024

arXiv 2024

[7] [7]

Proof-of-learning: Definitions and practice

Hengrui Jia, Mohammad Yaghini, Christopher A Choquette-Choo, Natalie Dullerud, Anvith Thudi, Varun Chandrasekaran, and Nicolas Papernot. Proof-of-learning: Definitions and practice. In 2021 IEEE Symposium on Security and Privacy (SP) , pages 1039–1056. IEEE, 2021

work page 2021

[8] [8]

OpenSkill: A faster asymmetric multi-team, multiplayer rating system

Vivek Joshy. Openskill: A faster asymmetric multi-team, multiplayer rating system. arXiv preprint arXiv:2401.05451, 2024

work page Pith review arXiv 2024