{"id":"3e1ea449-072e-4938-aa2e-ede59fbe7f7a","arxiv_id":"2607.13332","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Agora is a protocol that trained an 8.6B-parameter LLM on 500B tokens using churning, heterogeneous, internet-connected consumer GPUs at ~63% of centralized H100 token-per-FLOP efficiency.","lead":"A research group trained an 8.6-billion-parameter language model over 40 days using about 330 volunteer consumer GPUs connected by ordinary internet links, reporting ~63% of the token-per-FLOP efficiency of a centralized H100 run and convergence close to a centralized reference. The result is significant if it holds — it sketches a path to collective, permissionless frontier pretraining — but the paper's own headline compute numbers are internally inconsistent and its efficie","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Subspace-network compression (Sec 4.1) is validated only against other subspace-constrained runs; without a dense baseline, quality degradation and the 63% efficiency claim are unsubstantiated.","rationale":"The reader's verdict is CONDITIONAL, and I concur. The shared-subspace premise is indeed the load-bearing component: it enables the ~100× communication compression (Eq. 2) and the parameter reduction that underlies the efficiency metric. The paper's validation strategy cannot detect a quality penalty from the subspace itself, because every comparison (Figures 38, 19, 20) is between two runs that share the same subspace constraint. The central reference (Section 5.5) is explicitly modified to use the same reparameterized SSN and SPARTA, so it controls for decentralization but not for the architectural change. This is scientifically honest, but it means the headline claims of 'near-centralized convergence' and '63% of baseline efficiency' are conditional on the subspace being behavior-preserving — an assumption that is neither proven nor commonplace. A rank-51 subspace at d=5120 is a severe capacity restriction, especially for the token embedding, where all 128,256 token vectors are confined to a 51-dimensional space. The paper never specifies how U_k is chosen or whether it is learned; if fixed, the model's representational power is arbitrarily limited. The efficiency comparison to dense Llama 3.1 8B (Section 5.4) additionally conflates architecture with system: since the subspace model does fewer FLOPs per token, a higher tokens/TFLOP is expected regardless of system overhead. A model-matched centralized baseline (the same subspace model trained on H100s) would be the correct comparator for the 63% figure, and a dense baseline is needed to assess the subspace penalty. Both are missing. These are fixable concerns — the systems demonstration itself is credible — so the CONDITIONAL verdict stands, with the condition being the addition of a dense-baseline comparison (or explicit acknowledgment that the model is not competitive with dense pretraining).","tokens_in":49195,"tokens_out":11836,"duration_ms":128844,"concrete_test":"Train a centralized, dense-parameterization version of Pluralis-8B (full-rank output projections, no subspace embedding) on the same FineWeb-Edu data for the same 60k steps/500B tokens with the identical optimizer and LR schedule, and compare final training loss and a standard benchmark suite (e.g., HellaSwag, MMLU, WinoGrande). If the dense model's loss is >0.1 nats lower or benchmarks are >2 points higher, the subspace premise fails to be behavior-preserving at 8B scale. Report also how U_k is initialized and whether it is trained; if it is fixed, repeat with a learned U_k to isolate the basis choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—near-centralized convergence at 63% of H100 baseline efficiency—rests on the shared-subspace premise of Section 4.1. Eq. (3) reparameterizes every output projection and the token embedding as W = Z U_k^T, confining the entire residual stream to a common k-dimensional subspace (k≈51 at d=5120). This is what compresses inter-stage traffic ~100× and reduces model FLOPs per token, making pipeline parallelism viable at internet bandwidth. The premise is untested against an unconstrained dense model: the 1B ablations (Fig. 38) and the central reference (Sec. 5.5) all use the same subspace constraint, so they cannot reveal quality loss from the subspace itself. Two specific weaknesses: (1) U_k appears to be a fixed, unspecified basis; if it is not learned, the model's capacity is confined to an arbitrary low-dimensional space, likely degrading performance at scale. (2) The efficiency comparison to a dense Llama 3.1 8B baseline (Sec. 5.4) is not model-matched: the subspace model does fewer FLOPs/token, so a higher tokens/TFLOP is expected even with identical system efficiency. The 63% figure therefore conflates architecture with system. If the subspace constraint causes even a modest loss penalty, the 'within a small margin of a centralized reference' claim is misleading because the reference inherits the same penalty.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Agora, a protocol and system for training large language models over a permissionless, heterogeneous, internet-connected pool of GPUs. It combines pipeline-parallel model sharding with reparameterized subspace networks (SSNs) to compress inter-stage activations/gradients, and asynchronous sparse weight averaging (AsyncSPARTA) to keep data-parallel replicas in consensus without per-step gradient all-reduce. The authors report a 40-day, 500B-token pretraining run of an 8.6B-parameter model, Pluralis-8B, on FineWeb-Edu, using ~330 contributor nodes with continuous churn, sustaining ~170k tokens/s and 4.2 tokens/TFLOP of pooled compute, which they state is 63% of a centralized H100 baseline, with training loss converging close to a centralized reference run. The paper also includes system design details, fault-tolerance mechanisms, a scaling analysis, and 1B-scale ablations on convergence robustness.","tokens_in":49442,"tokens_out":4416,"duration_ms":43370,"significance":"If the central claims hold, this is a significant systems contribution: it would be the first demonstration that a multi-billion-parameter LLM can be pretrained at scale over preemptible, individually owned, heterogeneous GPUs connected by the internet, with near-centralized convergence and modest efficiency loss. The engineering is substantial: fault-tolerant sparse all-reduce, admission control and sync phases, deterministic backward pinning, and a load-balanced pipeline over DHT-based peer discovery. The paper also makes code and a live dashboard available, which is a strength for reproducibility. However, the ML-level validation is currently incomplete: the subspace compression that underpins the system is validated only against other subspace-constrained runs, and the efficiency comparison is not model-matched. These gaps are load-bearing for the paper's headline claims and require either additional experiments or a careful restatement of the claims.","major_comments":[{"comment":"The claim that Pluralis-8B 'converged to within a small margin of a centralized reference' is not validated against a dense, unconstrained transformer. Equation (3) reparameterizes every output projection and the token embedding as W = Z U_k^T, confining the residual stream to a rank-k subspace (k≈51 at d=5120). All ablations and the central reference in Section 5.5 use the same subspace constraint, so they cannot reveal quality loss caused by the constraint itself. Because this premise is what makes the ~100× inter-stage compression and the entire pipeline-parallel design viable, a comparison to a dense baseline—or at least a small-scale dense run under Agora—is needed to separate architecture-induced degradation from system-induced degradation. Without it, the 'small margin' claim is incomplete.","section":"§4.1, §5.5, Fig. 38"},{"comment":"The 63% efficiency comparison is not model-matched. The centralized baseline is a dense Llama 3.1 8B trained with FSDP2 on H100s, whereas Pluralis-8B's reparameterized SSN has fewer trainable parameters (~8.6B vs ~12B dense-equivalent) and performs fewer FLOPs per token because output projections are low-rank. A higher tokens/TFLOP is therefore expected even at identical system efficiency, so the 63% figure conflates architectural FLOP reduction with system efficiency. Additionally, §5.1 reports 'measured 30.8 ZettaFLOPs (3.08×10^22)' for 500B tokens over 40 days, which implies ~16.2 tokens/TFLOP of actual compute, while Table 4 reports 4.31 tokens/TFLOP using C_pool = 39,749 TFLOP/s spec-sheet capacity. These two metrics should be reconciled; as written, the abstract's efficiency claim is ambiguous.","section":"§5.4, Table 4, §5.1"},{"comment":"AsyncSPARTA's convergence relies on the delay-corrected sparse averaging scheme, but the convergence guarantee is only cited to 'Maziane et al. (2026)' with no statement of assumptions or proof sketch. Given that the run's convergence under asynchronous averaging with partial all-reduce participation is a central empirical claim, the paper should at least state the conditions under which the guarantee holds (e.g., smoothness, bounded delay, averaging cadence) or provide a proof in an appendix. As it stands, the convergence argument is an appeal to a same-group reference that is not publicly identifiable.","section":"§4.2, Eq. (8), §4.2.2"},{"comment":"The scaling analysis concludes that communication constraints relax as models grow, but this relies on the assumption that the required subspace rank grows more slowly than width. The paper explicitly says 'We do not claim a scaling law from two configurations' and reports only rank 40 at d=2048 and rank ≈51 at d=5120. With two points, the trend is not established. Since the introduction and conclusion frame Agora as a path to frontier-scale training, this extrapolation should be either supported with more data or clearly labeled as a conjecture rather than a demonstrated property.","section":"§3.4.5, §4.1"}],"minor_comments":[{"comment":"The notation for the basis U_k is inconsistent: Eq. (2) sends (X - PE - T_fixed) U_k and reconstructs with U_k^T, while Eq. (3) writes W = Z U_k^T. Clarify whether columns of U_k are the orthonormal basis and ensure the row/column convention is used consistently throughout.","section":"§4.1, Eq. (2)–(3)"},{"comment":"The centralized reference run is only 27k iterations, while Pluralis-8B runs for 60k steps. The paper should justify why a shorter reference is sufficient to support the 'small margin' convergence claim, especially since training dynamics change over the full schedule.","section":"§5.5"},{"comment":"The scaling analysis says 'the rank we can use appears to grow more slowly than the width,' but the rank is described as a hyperparameter. Is U_k fixed and unlearned, or is it trained/updated? If fixed and arbitrary, this is a strong architectural assumption that should be stated explicitly and tested.","section":"§3.4.5, §6"},{"comment":"The term 'Unextractable Protocol Models (UPMs)' is introduced in the abstract/introduction but never defined or used in the system description or experiments. Either define the concept and its role, or remove it from the paper.","section":"Abstract, §1"},{"comment":"Several core methods are cited to unpublished or same-group references without arXiv IDs or version information (Ramasinghe et al. 2025; Beton et al. 2025; Ajanthan et al. 2026; Long et al. 2025; Maziane et al. 2026). Full citations should be added so readers can verify the methods.","section":"Throughout"},{"comment":"Typo: 'SW ARM parallelism' should be 'SWARM parallelism' (Section 3.2). Also check formatting of author names (e.g., 'V olz and Brazil, 2018').","section":"§2.2, §3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper reports an impressive systems achievement, but the two central quantitative claims—near-centralized convergence and 63% of centralized efficiency—are not yet supported by the right comparisons. The most important missing experiment is a dense-baseline comparison at some scale. If the authors can provide even a small-scale dense-vs-subspace comparison and a model-matched efficiency number, the paper would be much stronger. I also note the heavy reliance on same-group references without arXiv IDs; this should be addressed in revision, and the editor may want to check for concurrent/prior work overlap."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this paper does something new that is actually worth engaging with — an 8.6B-parameter model trained over 40 days on 500B tokens by ~330 heterogeneous, churning, internet-connected consumer GPUs, with detailed telemetry and a loss curve that tracks a centralized reference. If that claim holds up, it genuinely broadens who can do frontier-scale training. The paper deserves a serious referee, but not without revisions.\n\nWhat's good: the systems engineering is substantial and unusually transparent. Pseudocode, configuration tables, admission dynamics, failure telemetry, per-stage communication/compute breakdowns, and 1B-scale ablations — this is a real experimental report, not a sketch. The authors disclose the caveats themselves: the singleton Authorizer, operator-only head/tail stages, screened admission, and the 60-node cap. That honesty counts. The reparameterized subspace network and the delta-rule AsyncSPARTA integration are concrete contributions that others can build on. The ablation on pinning the backward pass to the same worker is a nice, clean result.\n\nWhere it gets soft. First, the arithmetic doesn't work as printed. The abstract reports 4.2 tokens/TFLOP over 500B tokens, which implies roughly 119 ZettaFLOPs, but Section 5.1 says 30.8 ZettaFLOPs. That's a factor of ~4 gap, and it's not some minor rounding issue — it's the headline efficiency claim. The authors need to reconcile which FLOP base they're using (dense-equivalent vs. subspace FLOPs) and report consistently.\n\nSecond, the 63%-of-H100 efficiency comparison is not model-matched. The baseline is a dense Llama 3.1 8B with FSDP2; the Pluralis model does fewer FLOPs per token because of the subspace compression. So a higher tokens/TFLOP is expected even at identical system efficiency. The 63% conflates architecture with system. A dense baseline of the same model size or a FLOP-matched comparison is needed.\n\nThird, and relatedly, the subspace constraint is validated only against other subspace-constrained runs — including the centralized reference. There is no unconstrained dense baseline, so we cannot see whether the subspace itself costs quality. The stress-test note makes this point, and it's right. It's not fatal to the existence claim: the run still demonstrates that this architecture trains stable over internet GPUs. But it does mean \"within a small margin of a centralized reference\" is circular — the reference inherits the same constraint.\n\nFourth, the core methods are cited to the authors' own 2025-2026 papers without arXiv IDs, so external verification is hard. That's not a fatal flaw here because the paper describes the methods in enough detail to be reproduced, but it's a burden the reviewers should push on.\n\nBottom line: this is a systems paper with a real, important existence proof. The efficiency and loss-comparison claims need to be reframed or fortified, but the core result — permissionless, internet-scale pretraining at meaningful efficiency — is credible and should not be dismissed. I'd send it to a serious systems/ML venue with strong reviewers, and I'd require the FLOP arithmetic fixed and a dense baseline for the subspace penalty before acceptance.","headline":"A credible existence proof for permissionless internet-scale pretraining, with real but fixable inconsistencies in the headline efficiency numbers.","tokens_in":50217,"tokens_out":2470,"would_cite":true,"duration_ms":27857,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agora claims multi-billion-parameter models can be pretrained permissionlessly over the internet by churning consumer GPUs, and demonstrates an 8.6B, 500B-token run at ~63% of a centralized H100 baseline's efficiency.","keywords":["protocol learning","distributed pretraining","pipeline parallelism","subspace networks","sparse weight averaging","fault-tolerant all-reduce","permissionless training","consumer GPU training"],"falsifier":"Train the same 8.6B architecture on the same 500B tokens in a datacenter without the subspace constraint and compare final training loss with 2.41; if the unconstrained model is materially better, the compression premise fails. Alternatively, measure how the minimal usable subspace rank scales with hidden dimension: if rank grows proportionally to d instead of sublinearly (40 at d=2048 vs ~51 at d=5120), the 100x compression and the bandwidth-adequacy argument collapse at frontier scale.","tokens_in":48906,"feed_emoji":"🌐","tokens_out":7781,"duration_ms":74870,"temperature":0.7,"pith_summary":"Large-model pretraining has been confined to datacenters because its communication primitives assume high bandwidth, low latency, homogeneous accelerators, and a stable owner. Agora replaces those assumptions with a protocol: the model is split into pipeline stages, inter-stage traffic is compressed roughly 100x by confining activations to a low-rank subspace, and replica consensus is kept by asynchronously averaging a small fraction of weights every 20 steps instead of synchronizing gradients. On this machinery the paper trained Pluralis-8B, an 8.6B-parameter model on 500B tokens, over 40 days with roughly 330 heterogeneous consumer-GPU contributors that joined and left freely, sustaining about 170k tokens/s and about 63% of a centralized H100 baseline's tokens-per-TFLOP efficiency while converging to within a small margin of a centralized reference. If correct, this means the unused GPU pool outside datacenters can be pooled permissionlessly for frontier-scale pretraining, with no single participant ever holding the full weights. The load-bearing question is whether the subspace confinement that enables this remains quality-preserving as models grow.","feed_headline":"330 consumer GPUs pretrain an 8.6B model over the internet","feed_subtitle":"A 40-day, 500B-token run hit ~63% of a datacenter's efficiency while nodes joined and left at will.","key_machinery":"The system rests on two communication-reduction mechanisms. First, reparameterized subspace networks: every layer's output projection and the trainable token embedding are factored through a fixed orthonormal basis, so all hidden states and gradients crossing a pipeline boundary live in a k-dimensional subspace (rank roughly 51 at hidden dimension 5120); a stage sends n×k coordinates per token instead of n×d, a roughly 100x cut in inter-stage traffic. Second, AsyncSPARTA: replicas of a stage take local optimizer steps and, every 20 steps, all-reduce a rotating 5% slice of their weights in the background rather than exchanging gradients before every step; a delta rule preserves each replica's","core_discovery":"The paper reports the first full-scale demo of its protocol: an 8.6B-parameter decoder-only transformer with 32 layers sharded into seven pipeline stages, trained for 40 days on 500B tokens of an education-filtered web corpus. The compute pool mixed operator-run nodes with 330 external contributor nodes, predominantly consumer GPUs on internet links ranging from about 200 Mbit/s to several Gbit/s with up to 80 ms round-trip latency; 669 nodes joined and 607 departed during the run. The run sustained about 170k tokens/s, reached a steady-state 4.2 tokens per TFLOP of pooled compute (63% of a centralized H100 baseline), and converged to a final training loss of 2.41, close to a centralized ref","pith_inferences":["If the observed sublinear rank growth (rank 40 at d=2048, rank ~51 at d=5120) continues, the compression ratio improves with scale; if rank instead grows proportionally to width, the 100x compression and the admission-bandwidth floor would both need to be revisited.","Because latency, not bandwidth or raw GPU FLOPs, governed contributor throughput in the reported measurements, a natural extension not tested here is geographic placement of trainers and reducers near contributors to shorten round trips.","The 200 Mbit/s / 80 ms admission floor means the practical 'permissionless' pool is limited to well-connected machines; reach would grow if the compression or averaging could tolerate slower residential links.","The convergence comparison is against a reference that also uses subspace compression and sparse averaging, so the run demonstrates decentralized execution matching centralized execution of the same compressed architecture — not that the compressed architecture matches an uncompressed one."],"forward_implications":["Pretraining compute can be pooled from individually owned, geographically distributed, preemptible GPUs: the run held near-170k tokens/s through 669 joins and 607 departures, with about 60 contributors admitted at a time.","The 63%-of-datacenter efficiency and near-centralized convergence indicate that the two communication cuts — subspace-compressed pipeline traffic and sparse asynchronous averaging — are enough to keep internet-grade links off the training critical path at the 8B scale.","No node ever holds the full model: workers hold one pipeline stage and trainers hold none, so the trained model is collectively owned; the paper argues this opens an economically sustainable route to open frontier training.","The paper's scaling analysis, which treats compression rate as fixed in the pessimistic case, indicates that bandwidth and latency constraints relax as hidden dimension and depth grow, so the protocol is claimed to become easier, not harder, to sustain at larger scale."],"fun_headline_variants":["Internet-distributed LLM: 330 GPUs train 8.6B model","Agora: Pretrain large models using home GPUs, no datacenter","8.6B model trained on 500B tokens by 330 volunteer GPUs","Consumer GPUs hit 63% of H100 efficiency in collective run","Protocol Learning: Permissionless collective pretraining goes live"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that confining every layer's output projection and the token embedding to a common low-dimensional subspace (rank ~51 at hidden dimension 5120) preserves model quality at scale; the paper validates this only against a reference run that also enforces the constraint and explicitly declines to claim a scaling law from two configurations.","fun_headline_variants_meta":{"raw":{"variants":["Internet-distributed LLM: 330 GPUs train 8.6B model","Agora: Pretrain large models using home GPUs, no datacenter","8.6B model trained on 500B tokens by 330 volunteer GPUs","Consumer GPUs hit 63% of H100 efficiency in collective run","Protocol Learning: Permissionless collective pretraining goes live"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000454,"raw_usage":{"total_tokens":2183,"prompt_tokens":875,"completion_tokens":1308,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":1207}},"tokens_in":619,"tokens_out":1308,"duration_ms":13073,"temperature":1.0,"reasoning_tokens":1207,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T05:30:19.076159+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 8.6B architecture on the same 500B tokens in a datacenter without the subspace constraint and compare final training loss with 2.41; if the unconstrained model is materially better, the compression premise fails. Alternatively, measure how the minimal usable subspace rank scales with hidden dimension: if rank grows proportionally to d instead of sublinearly (40 at d=2048 vs ~51 at d=5120), the 100x compression and the bandwidth-adequacy argument collapse at frontier scale.","supporting_citations":[],"review_version":1}