Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

NoLoCo: No-all-reduce Low Communication Training Method for Large Models

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes NoLoCo, a modified Nesterov-momentum optimizer that synchronizes model weights only between randomly chosen pairs of replicas, and argues this removes the need for all-reduce and all collective communication while…

desk verdict A plausible no-all-reduce training method with a load-bearing proof error and an unreported key hyperparameter; worth reviewing but not yet convincing. read the letter →

arxiv 2506.10911 v1 pith:IH3TBO5T submitted 2025-06-12 cs.LG

Jari Kolehmainen , Nikolay Blagoev , John Donaghy , Oğuzhan Ersoy , Christopher Nies This is my paper
classification cs.LG
keywords NoLoColow-communicationtrainingdecentralizeddistributedoptimizationNesterovmomentumpipelineparallelismall-reducelargelanguagemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Training large language models normally assumes a tightly connected cluster because data-parallel training must keep all replicas synchronized. NoLoCo is an optimizer designed to break that assumption: it never performs an all-reduce or any global collective operation, instead implicitly synchronizing model weights by partially averaging each replica's slow weights with one randomly chosen other replica at each outer step. The paper argues that this modified Nesterov momentum update keeps replicas from diverging, and it reports that models from 125M to 6.8B parameters converge comparably to, and up to 4% faster than, the low-communication DiLoCo baseline on two language modeling datasets. If the claim holds, large-model training could work over low-bandwidth, high-latency networks instead of dedicated supercomputing fabrics, because the synchronization cost per step grows only logarithmically with worker count and no global barrier idles fast workers.

What carries the argument

The load-bearing object is the modified Nesterov momentum outer update of Eq. (2): $\delta_{t,i} = \alpha\delta_{t-1,i} - (\beta/n)\sum_j \Delta_{t,j} - \gamma(\phi_{t,i} - (1/n)\sum_j \phi_{t,j})$. The third term is what turns pairwise averaging into implicit synchronization: each worker's slow weights are dragged toward the random subgroup mean, and because the subgroup is re-sampled every outer step, information spreads across all workers without any collective call. The convergence argument studies the standard quadratic stochastic loss $L(\theta)=\frac12(\theta-c)^T A(\theta-c)$ with $c\sim\mathcal{N}(0,\Sigma)$, which reduces the expected-value recursion to a linear recurrence whose roots stay below 1 when $\beta>\alpha$ and $0<\omega\Lambda_i\le 1$; the variance recursion then gives $\mathrm{Var}(\phi_{t,i})\propto\omega^2$, so the inner learning rate controls replica divergence. Random pipeline routing is the second mechanism: by letting a stage receive inputs from any replica of the previous stage, it mixes weights across data-parallel pipelines and is reported to lower replica variance by up to 15% at a small cost in validation loss.

What would settle it

Measure the actual variance of $\phi_{t,i} - (1/n)\sum_j \phi_{t,j}$ during training and compare it with Conjecture 1's prediction $2((n-1)/n)^2\mathrm{Var}(\phi_{t,i})$; if the mismatch does not shrink as the number of data-parallel workers grows, the proof's independence assumption is doing the work and the theorem would not transfer to the correlated setting. A second, direct test is a real geo-distributed run with hundreds of accelerators: if wall-clock time per outer step does not drop by roughly $\log_2(n)$ relative to a tree all-reduce and validation perplexity does not track the reported gap to DiLoCo, the practical speed-up claim fails.

Watch

Extended reading notes

Core claim

The central claim is that explicit all-to-all synchronization is not necessary for low-communication large-model training. In NoLoCo, the outer update is $\delta_{t,i} = \alpha\delta_{t-1,i} - (\beta/n)\sum_j \Delta_{t,j} - \gamma(\phi_{t,i} - (1/n)\sum_j \phi_{t,j})$, where $\Delta_{t,i} = \theta_{t+1,i} - \phi_{t,i}$ is the local outer gradient, $\alpha$ is momentum, $\beta$ the outer learning rate, and $\gamma$ the local averaging strength, with the group of size $n=2$ re-sampled randomly each outer step. The third term pulls each worker's slow weights toward the group average, so weights are implicitly synchronized even though no accelerator ever talks to all others. On a quadratic stochastic loss the paper proves the expected slow weights converge to the optimum and the variance across replicas is proportional to the square of the inner learning rate, which lets a cosine learning-rate schedule drive eventual consistency. Empirically, NoLoCo matches or beats DiLoCo in validation perplexity across model sizes and worker counts while requiring no global blocking communication, replacing a tree all-reduce that costs about $2t_c\log_2(n)$ with pairwise averaging that costs about $2t_c$.

Load-bearing premise

The convergence proof assumes that the model copies behave as statistically independent replicas, even though the authors acknowledge that the routing and previous averaging steps make them correlated, so the theoretical guarantee rests on that approximation being good enough.

Editorial extensions

If this is right

  • The outer synchronization cost per step drops from about $2t_c\log_2(n)$ for a tree all-reduce to about $2t_c$ for pairwise averaging, so the advantage grows with worker count and with message-latency variance.
  • Removing the global barrier eliminates the idling where fast workers wait for slow ones; the paper's simulations put the total-time saving near 20% at 1024 accelerators with 100 inner steps per outer step.
  • The replicas remain an ensemble rather than identical copies, and the standard deviation across replicas tracks the inner learning rate (empirical Pearson correlation between 0.91 and 0.97), so a learning-rate schedule can be used to drive eventual consistency to arbitrary numerical accuracy.
  • NoLoCo's final validation perplexity is a few percent worse than fully sharded data parallel training but up to 4% better than DiLoCo, with the gap to fully synchronized training shrinking for larger models and smaller data-parallel world sizes.
  • Random pipeline routing alone, without any outer synchronization, reduces replica weight variance by up to 15% while raising validation perplexity by up to 4%, and this tradeoff becomes less pronounced for larger models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if pairwise random averaging is enough for convergence, the same principle could be combined with gradient compression or asynchronous updates, since correctness no longer depends on a global barrier.
  • Beyond the paper: the claimed wall-clock advantage rests on a log-normal latency model rather than a measured geo-distributed deployment, so the natural next experiment is a real run across distant accelerators, which would also reveal how often a random pair lands on a slow link.
  • Beyond the paper: the independence approximation in the proof gets better as the data-parallel world size grows, so the theory predicts the method becomes more reliable exactly in the regime where all-reduce is most expensive; testing convergence at hundreds or thousands of workers would probe this.
  • Beyond the paper: if replica variance is truly controlled by the inner learning rate, a learning-rate schedule could be tuned to make the final ensemble agree to arbitrary precision without any explicit synchronization step, giving a parameter-server-free path to a single merged model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes NoLoCo, a distributed training method that eliminates collective communication (all-reduce) by performing outer-optimizer synchronization between randomly chosen pairs of replicas and by using random pipeline routing, with a modified Nesterov momentum update that includes a local averaging term. The authors provide a convergence analysis for a quadratic loss (expected value tends to zero, variance proportional to inner learning rate squared) and report language-model experiments on 125M-6.8B models comparing NoLoCo to DiLoCo and FSDP. They claim up to 4% faster convergence than DiLoCo and estimate an order-of-magnitude reduction in synchronization cost. The manuscript includes source code, an explicit latency model, and ablations of random routing and batch size.

Significance. If the central claims were established, NoLoCo would be a meaningful advance for low-bandwidth, high-latency distributed training: removing global blocking all-reduce and replacing it with point-to-point pair averaging is an appealing design, and the theoretical variance-control prediction gives a concrete mechanism for eventual consistency. The paper is candid about several limitations (suboptimal hyperparameters, latency model vs measured deployment, and the independence assumption). However, the theoretical guarantee is not currently established: the expectation recurrence has a sign error, the covariance term in the variance recurrence has an incorrect power of gamma, and the proof relies on an admitted-false independence assumption and an unproven conjecture. The empirical support for the headline speed-up is weakened by single-run comparisons with untuned hyperparameters and no error bars. These issues are fixable in revision, so the underlying idea remains worth pursuing.

major comments (5)
  1. [Appendix A.2, Eq. (32)] The expectation recurrence has a sign error. From Eq. (2)/Eq. (31), delta_t,i = alpha delta_{t-1,i} - (beta/n) sum_j Delta_{t,j} - gamma(phi_t,i - (1/n) sum_j phi_t,j). Taking expectations, the gamma term vanishes by Lemma 1, so E(delta_t,i) = alpha E(delta_{t-1,i}) - beta E(Delta_{t,i}), not the plus sign used in Eq. (32). This sign propagates through Eqs. (33)-(43), and the matrix D in Eq. (43) should be I + alpha I - beta(B^m - I), not I + alpha I + beta(B^m - I). Consequently the stability condition 'beta > alpha' below Eq. (53) is derived from the wrong recurrence. The proof of Theorem 2 must be re-derived with the correct sign; this is load-bearing for the paper's central convergence claim.
  2. [Appendix A.3, Eq. (55)] The covariance approximation contains an incorrect power of gamma. The only direct dependence of delta_t,i on phi_t,i in Eq. (31) is the term -gamma(phi_t,i - (1/n) sum_j phi_t,j). Under the text's stated neglect of covariances with Delta and delta_{t-1}, the leading term is Cov(phi_t,i, delta_t,i) approximately -gamma((n-1)/n)V(phi_t,i), linear in gamma, not -gamma^2((n-1)/n)V as written. This error changes the coefficient of the second term in Eq. (62), the definition of d_V in Eq. (69), and hence the stability window for gamma in Eq. (74). As written, Theorem 3's variance bound is not derived; the corrected recurrence should be solved and the stability condition updated.
  3. [Appendix A.2, Conjecture 1] The convergence proof relies on assumptions that are acknowledged in the text to be false or unproven. In Appendix A.2 the authors state that slow weights across replicas are 'generally not independent' and yet assume independence for the convergence analysis; Conjecture 1 for the variance of the deviation from the group mean is stated without proof. The text asserts that the error 'will become smaller when the data parallel world size becomes larger' but provides no quantitative bound. Since the variance recurrence (Eqs. (56)-(70)) and Theorem 3 depend on Conjecture 1, the guarantee is conditional on an unverified assumption. Please either prove the conjecture under a stated weaker condition, or provide a bound on the error and state the theorem as conditional.
  4. [Section 4, Eq. (2)] The hyperparameter gamma is never reported. Eq. (2) includes gamma, and the theoretical stability condition (Eq. (74)) is a constraint on gamma. Section 4 reports alpha=0.5, beta=0.7, and group size n=2 for NoLoCo but does not state the gamma value used in any run. Without this value the reader cannot check whether the experiments operate in the claimed stable regime, and the experimental configuration is not reproducible. Please report gamma for every configuration and, ideally, provide an ablation over gamma around the predicted stability boundary.
  5. [Section 5.1, Table 2] The empirical comparison supporting the headline 'up to 4% faster convergence' is not backed by uncertainty estimates. Table 2 and Fig. 2 report single training runs without error bars or repeated seeds, and Section 5.1 states that the batch size and learning rate are taken from an FSDP-optimized study and are 'likely sub-optimal' for both DiLoCo and NoLoCo. The differences between DiLoCo and NoLoCo in Table 2 are often small (e.g., 26.8 vs 26.4 for the small model on Reddit with DP=8) and could be within run-to-run noise. Please provide multiple seeds with error bars (or at least report the number of runs) and either tune hyperparameters for both methods or clearly label the results as a preliminary comparison rather than evidence of faster convergence.
minor comments (6)
  1. [Section 5.1] There is a typo: 'NiLoCo' should be 'NoLoCo' in the sentence 'we observe that NiLoCo is slightly better than DiLoCo'.
  2. [Fig. 3B] The y-axis label sigma/sigma_max is not defined in the text; please state explicitly whether sigma is the standard deviation of a single parameter, averaged over all parameters, and how sigma_max is computed.
  3. [Eqs. (45)-(46)] The characteristic equations involve matrix square roots and matrix inequalities (e.g., '0 < r_2 <= r_1 <= D') that are not well-defined for general matrices; these should be stated in terms of the eigenvalues of D or of A.
  4. [Eq. (24)] There appears to be an index inconsistency in the summation: the term B^{k-1} appears where B^k is used elsewhere; please check and correct the indexing.
  5. [Appendix A.1, Eq. (21)] In Eq. (21), the variance of a sum of gradient terms is approximated by the sum of variances without a justification for neglecting covariances of non-consecutive fast weights; please make this approximation explicit and, if possible, bound the neglected terms.
  6. [Section 5.2] The random-routing ablation is presented with ratios (sigma/sigma_base, L/L_base) but without error bars; given the small effects (10-15%), please clarify how many runs these ratios are based on.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NoLoCo's no-all-reduce operation is a protocol consequence, and the convergence analysis is a self-contained derivation for the stated quadratic loss; the documented concerns (unproven Conjecture 1, unreported gamma, and the apparent gamma-scaling issue in Eq. 55) are correctness risks, not circularity.

full rationale

The paper's central communication claim is not derived by fitting: NoLoCo avoids all-reduce because Eq. 2 restricts outer synchronization to randomly chosen pairs and Eq. 3 updates only local weights; this is a protocol definition, not a prediction extracted from the results, so no self-definitional reduction is present. The convergence proof in Appendix A is likewise self-contained: it starts from the stated quadratic loss (Eq. 8), assumes SGD inner steps (Eq. 10), and propagates means and variances through the explicit update rules (Eqs. 2-3, 18-30, 31-53). It does not assume the conclusion; Lemma 1 is proved by induction in Appendix B, and Conjecture 1 is explicitly labeled an assumption rather than an imported theorem. The independence assumption is confessed to be false, which weakens the proof but does not make it circular. Section 5.1's correlation between replica standard deviation and learning rate is an empirical check of the predicted V proportional to omega squared scaling using the same training runs; this is weak self-support but not a fitted parameter renamed as a prediction. The only self-citation (Gourav et al., 2024, in a list of speech works) is not load-bearing, and no uniqueness theorem from prior work by these authors is invoked. The skeptical concern about Eq. 55 (covariance linear in gamma, not gamma squared) and the absence of reported gamma values would affect the validity of Theorem 3's stability window, but these are mathematical correctness issues, not instances where an output equation reduces to an input by construction.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central derivations rest on a simplified quadratic loss, an SGD inner optimizer, and an independence assumption for slow weights that the authors admit is false. The latency benefit rests on a log-normal communication model. Hyperparameters such as gamma, alpha, beta, m, n and inner learning rates are chosen rather than derived.

free parameters (6)
  • gamma (local weight averaging parameter)
    Appears in Eq. 2 as the coefficient of the slow-weight difference term; central to preventing weight divergence, but no value is reported in the paper.
  • alpha (Nesterov momentum) = 0.5 for NoLoCo, 0.3 for DiLoCo
    Chosen by the authors; no sensitivity analysis is reported, and the values differ between the compared methods.
  • beta (outer learning rate) = 0.7
    Shared with the DiLoCo baseline and taken from prior practice, not tuned for NoLoCo.
  • m (inner steps per outer step) = 50 for NoLoCo, 100 for DiLoCo
    Chosen to keep communication low; the paper notes NoLoCo still communicates less than DiLoCo with doubled frequency.
  • n (group size) = 2
    Minimum group size, selected to minimize communication and used in all experiments.
  • inner learning rates = 0.0006 (125M), 0.0002 (1.3B), 0.00012 (6.8B)
    Taken from OPT hyperparameters; the authors acknowledge they are likely suboptimal for the decentralized methods.
assumptions (6)
  • domain assumption The loss is quadratic: L(theta) = 1/2 (theta - c)^T A (theta - c) with c ~ N(0, Sigma), A positive definite.
    Appendix A.1: this simplified loss is standard for convergence analysis but does not model transformer training.
  • domain assumption The inner optimizer is stochastic gradient descent with a constant learning rate.
    Appendix A.1: used to derive Eqs. 11-12; real experiments use Adam with a cosine schedule.
  • ad hoc to paper Slow weights across replicas are identically distributed and independent.
    Appendix A.2: the authors state this is not true and introduces an error; it is needed for Lemma 1 and the convergence proof.
  • ad hoc to paper Conjecture 1: variance of (phi_t,i - 1/n sum_j phi_t,j) is approximately 2((n-1)/n)^2 Var(phi_t,i).
    Appendix A.2: stated without proof and used in the variance analysis.
  • domain assumption Covariances of non-consecutive fast weights are neglected.
    Appendix A.1, Eq. 26: the paper follows Zhang et al. in omitting these covariances.
  • domain assumption Communication times follow a log-normal distribution in the latency analysis.
    Section 5.3: used to simulate all-reduce versus local averaging speed-ups.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NoLoCo: No-all-reduce Low Communication Training Method for Large Models." pith.science (2026). https://pith.science/paper/IH3TBO5T

@misc{pith2026250610911,
  author       = {Pith},
  title        = {Pith review of: NoLoCo: No-all-reduce Low Communication Training Method for Large Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IH3TBO5T}},
  note         = {Machine review of arXiv:2506.10911}
}
abstract

Training large language models is generally done via optimization methods on clusters containing tens of thousands of accelerators, communicating over a high-bandwidth interconnect. Scaling up these clusters is expensive and can become impractical, imposing limits on the size of models that can be trained. Several recent studies have proposed training methods that are less communication intensive, avoiding the need for a highly connected compute cluster. These state-of-the-art low communication training methods still employ a synchronization step for model parameters, which, when performed over all model replicas, can become costly on a low-bandwidth network. In this work, we propose a novel optimization method, NoLoCo, that does not explicitly synchronize all model parameters during training and, as a result, does not require any collective communication. NoLoCo implicitly synchronizes model weights via a novel variant of the Nesterov momentum optimizer by partially averaging model weights with a randomly selected other one. We provide both a theoretical convergence analysis for our proposed optimizer as well as empirical results from language model training. We benchmark NoLoCo on a wide range of accelerator counts and model sizes, between 125M to 6.8B parameters. Our method requires significantly less communication overhead than fully sharded data parallel training or even widely used low communication training method, DiLoCo. The synchronization step itself is estimated to be one magnitude faster than the all-reduce used in DiLoCo for few hundred accelerators training over the internet. We also do not have any global blocking communication that reduces accelerator idling time. Compared to DiLoCo, we also observe up to $4\%$ faster convergence rate with wide range of model sizes and accelerator counts.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Not Every Sync Is Safe: Calibrated DiLoCo Scheduling for Shared AI Infrastructure

    cs.DC 2026-06 conditional novelty 6.0 of 10

    Matched-random merge placement is the right control for DiLoCo scheduling claims; only calibrated overlap plus EWMA forecasting beats it in real vLLM sidecar replay (6.54%→5.09% SLO violations).

Reference graph

Works this paper leans on

31 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [2]

    This appendix shows proof that the modified version converges to the vicinity of the real optima when the hyperparameters are chosen appropriately. We structure the proof as follows: this section gives background context and derives expected value and variance for the inner iterations weights; Section A.2 provides convergence proof for the expected value ...

    work page 2013

  2. [3]

    Communication-efficient language model training scales reliably and robustly: Scaling laws for diloco.arXiv preprint arXiv:2503.09799,

    Zachary Charles, Gabriel Teston, Lucio Dery, Keith Rush, Nova Fallen, Zachary Garrett, Arthur Szlam, and Arthur Douillard. Communication-efficient language model training scales reliably and robustly: Scaling laws for diloco.arXiv preprint arXiv:2503.09799,

    arXiv

  3. [7]

    Efficient training of large language models on distributed infrastructures: a survey.arXiv preprint arXiv:2407.20018,

    11 NoLoCo: No-all-reduce Low Communication Training Method for Large Models Jiangfei Duan, Shuo Zhang, Zerui Wang, Lijuan Jiang, Wenwen Qu, Qinghao Hu, Guoteng Wang, Qizhen Weng, Hang Yan, Xingcheng Zhang, et al. Efficient training of large language models on distributed infrastructures: a survey.arXiv preprint arXiv:2407.20018,

    arXiv

  4. [8]

    Accelerating large language model training with 4d parallelism and memory consumption estimator.arXiv preprint arXiv:2411.06465,

    Kazuki Fujii, Kohei Watanabe, and Rio Yokota. Accelerating large language model training with 4d parallelism and memory consumption estimator.arXiv preprint arXiv:2411.06465,

    arXiv

  5. [9]

    Multi-modal retrieval for large language model based speech recognition

    Aditya Gourav, Jari Kolehmainen, Prashanth Shivakumar, Yile Gu, Grant Strimel, Ankur Gandhe, Ariya Rastrow, and Ivan Bulyko. Multi-modal retrieval for large language model based speech recognition. InFindings of the Association for Computational Linguistics ACL 2024, pages 4435–4446,

    work page 2024

  6. [10]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  7. [11]

    Gossip learning as a decentralized alternative to federated learning

    István Hegedűs, Gábor Danner, and Márk Jelasity. Gossip learning as a decentralized alternative to federated learning. InDistributed Applications and Interoperable Systems: 19th IFIP WG 6.1 International Conference, DAIS 2019, Held as Part of the 14th International Federated Conference on Distributed Computing Techniques, DisCoTec 2019, Kongens Lyngby, De...

    work page 2019

  8. [12]

    Intellect-1 technical report.arXiv preprint arXiv:2412.01152,

    SamiJaghouar, JackMinOng, ManveerBasra, FaresObeid, JannikStraube, MichaelKeiblinger, ElieBakouch, Lucas Atkins, Maziyar Panahi, Charles Goddard, et al. Intellect-1 technical report.arXiv preprint arXiv:2412.01152,

    arXiv

Show all 31 references
  1. [13]

    Eager updates for overlapped communication and computa- tion in diloco.arXiv preprint arXiv:2502.12996,

    Satyen Kale, Arthur Douillard, and Yanislav Donchev. Eager updates for overlapped communication and computa- tion in diloco.arXiv preprint arXiv:2502.12996,

    arXiv

  2. [14]

    Branch-train-merge: Embarrassingly parallel training of expert language models.arXiv preprint arXiv:2208.03306,

    Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettle- moyer. Branch-train-merge: Embarrassingly parallel training of expert language models.arXiv preprint arXiv:2208.03306,

    arXiv

  3. [16]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434,

    arXiv

  4. [17]

    Ring attention with blockwise transformers for near-infinite context

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889,

    arXiv

  5. [18]

    Voxtlm: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks

    Soumi Maiti, Yifan Peng, Shukjae Choi, Jee-weon Jung, Xuankai Chang, and Shinji Watanabe. Voxtlm: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Sp...

    2024

  6. [19]

    Decoupled momentum optimization.arXiv preprint arXiv:2411.19870,

    12 NoLoCo: No-all-reduce Low Communication Training Method for Large Models Bowen Peng, Jeffrey Quesnelle, and Diederik P Kingma. Decoupled momentum optimization.arXiv preprint arXiv:2411.19870,

  7. [20]

    Audiopalm: A large language model that can speak and listen.arXiv preprint arXiv:2306.12925,

    Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen.arXiv preprint arXiv:2306.12925,

    arXiv

  8. [23]

    Seq1f1b: Efficient sequence-level pipeline parallelism for large language model training.arXiv preprint arXiv:2406.03488,

    Ao Sun, Weilin Zhao, Xu Han, Cheng Yang, Xinrong Zhang, Zhiyuan Liu, Chuan Shi, and Maosong Sun. Seq1f1b: Efficient sequence-level pipeline parallelism for large language model training.arXiv preprint arXiv:2406.03488,

    arXiv

  9. [24]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    arXiv

  10. [25]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

    arXiv

  11. [26]

    Understanding short-horizon bias in stochastic meta- optimization.arXiv preprint arXiv:1803.02021,

    Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta- optimization.arXiv preprint arXiv:1803.02021,

    arXiv

  12. [28]

    Llava-mini: Efficient image and video large multimodal models with one vision token.arXiv preprint arXiv:2501.03895,

    Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token.arXiv preprint arXiv:2501.03895,

    arXiv

  13. [29]

    Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068,

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068,

    arXiv

  14. [30]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592,

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592,

    arXiv

  15. [2013]

    Megatron- lm: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053,

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron- lm: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053,

    1909 arXiv

  16. [2018]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report.arXiv preprint arXiv:2503.20215,

    arXiv

  17. [2019]

    Tree attention: Topology-aware decoding for long-context attention on gpu clusters.arXiv preprint arXiv:2408.04093,

    Vasudev Shyam, Jonathan Pilault, Emily Shepperd, Quentin Anthony, and Beren Millidge. Tree attention: Topology-aware decoding for long-context attention on gpu clusters.arXiv preprint arXiv:2408.04093,

    arXiv

  18. [2020]

    Boosting asynchronous decentralized learning with model fragmentation.arXiv preprint arXiv:2410.12918,

    Sayan Biswas, Anne-Marie Kermarrec, Alexis Marouani, Rafael Pires, Rishi Sharma, and Martijn De Vos. Boosting asynchronous decentralized learning with model fragmentation.arXiv preprint arXiv:2410.12918,

    arXiv

  19. [2021]

    Diloco: Distributed low-communication training of language models

    Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low-communication training of language models. arXiv preprint arXiv:2311.08105,

    arXiv

  20. [2022]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122,

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122,

    arXiv

  21. [2023]

    Dipaco: Distributed path composition.arXiv preprint arXiv:2403.10616,

    Arthur Douillard, Qixuan Feng, Andrei A Rusu, Adhiguna Kuncoro, Yani Donchev, Rachita Chhaparia, Ionel Gog, Marc’Aurelio Ranzato, Jiajun Shen, and Arthur Szlam. Dipaco: Distributed path composition.arXiv preprint arXiv:2403.10616,

    arXiv

  22. [2024]

    A survey on mixture of experts

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint arXiv:2407.06204,

    arXiv

  23. [2025]

    Documenting large webtext corpora: A case study on the colossal clean crawled corpus

    Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.