Pith. sign in

REVIEW 3 major objections 6 minor 51 references

Model Parallelism With Subnetwork Data Parallelism

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Subnetwork Data Parallelism claims that giving each worker a fixed overlapping slice of the model—entire blocks or neurons—cuts per-GPU memory 28–60% while matching or improving validation loss and downstream accuracy under FLOP-matched…

desk verdict The memory-savings story is real and the B-SDP headline survives, but the FLOP-matched performance claim is unverified and inflated for Bb-SDP and N-SDP. read the letter →

arxiv 2507.09029 v5 pith:NMLUEL2B submitted 2025-07-11 cs.LG cs.AI

Vaibhav Singh , Zafir Khalid , Pietro Cagnasso , Edouard Oyallon , Eugene Belilovsky This is my paper
classification cs.LGcs.AI
keywords subnetworkdataparallelismdistributedtrainingmemory-efficientmaskedgradientsFLOP-matchedcomparisontransformerpre-trainingparameteraveragingstructuredsparsity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish a new axis of distributed training: instead of replicating the whole model on every GPU or splitting it into sequential pipeline stages, each worker trains a fixed, structurally complete subnetwork—a subset of transformer blocks, residual blocks, or neurons—and overlapping parameter assignments keep the pieces aligned through averaging. The central claim is that this costs almost nothing in model quality: under a FLOP-matched budget, where SDP runs more but cheaper steps so total FLOPs equal DDP, the method cuts per-device memory by 28–60% while matching or improving validation loss and downstream accuracy. The paper argues the memory saving is structural rather than compressive, because forward masking physically removes parameters, gradients, optimizer states, and activations in proportion to the active fraction $C$, and it shows the savings stack with sharding and activation checkpointing. If true, the practical payoff is that larger models or longer sequences fit on the same hardware, and training becomes viable on lower-bandwidth interconnects, because SDP exchanges only parameter/gradient traffic and never activations.

What carries the argument

The load-bearing object is a fixed binary masking matrix $m \in \{0,1\}^{n \times |J|}$: $m_{i,j}=1$ means worker $i$ trains coordinate $j$, so row $i$ is that worker's subnetwork and column $j$ lists the workers that share coordinate $j$. SDP specializes this to structured masks that remove whole neurons/channels (N-SDP) or whole residual/transformer blocks (B-SDP), leaving a full path from input to loss on every worker, and it synchronizes through the gated average $\bar{m}(g_1,\dots,g_n)_j = \frac{1}{c}\sum_{i=1}^n m_{i,j}(g_i)_j$, where $c$ is the column coverage. This single primitive makes per-worker memory and per-step communication scale roughly as $C\times$ DDP while eliminating cross-worker activation traffic. For backward masking, the forward pass is unmasked, so each worker's gradient is unbiased, and Theorem 3.1 bounds the SGD suboptimality by a term that grows with the Frobenius distance $\rho^2 = \|\frac{1}{n} m_{\mathrm{uni}} - \frac{1}{c} m\|_F^2$ between uniform averaging and the mask's averaging; the argument then treats FLOP-matching, roughly $1/C$ extra iterations, as the currency that converts slower per-iteration convergence into equal or better final loss.

What would settle it

Train the 1B LLaMA model with B-SDP at $C=0.69$ and with DDP for the same number of optimizer steps (or the same wall-clock time), using the same FineWeb token budget as the DDP baseline, and compare validation loss; if SDP's final validation loss is clearly worse than DDP's, then the reported parity is an artifact of the extra FLOP-matched iterations and the memory savings come at a real quality cost at fixed step count.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a model can be trained as a collection of overlapping subnetworks whose memory footprint scales with the active fraction $C$ rather than the full model size. In forward-masked SDP, the mask removes whole blocks or neurons from the forward pass, so parameters, gradients, optimizer accumulators, and activations are all materialized only for the active coordinates; in backward-masked SDP, the forward pass is computed on the full model but gradients and accumulators exist only for active coordinates, which keeps gradient estimates unbiased. The consistency of the global model is maintained by column overlaps and gated averaging, and the convergence analysis characterizes the price of the mask as the Frobenius distance $\rho^2$ to uniform averaging. The authors then use a FLOP-matched protocol—scaling iterations roughly inversely with $C$—and report Pareto dominance over DDP in both memory and validation loss at 500M and 1B LLaMA scale, with downstream benchmark averages within one point of DDP. On image classification, the same recipe matches or exceeds DDP accuracy at 40–60% of the memory, and at very sparse coverage the backward-masked variant avoids the collapse that forward-masked variants show.

Load-bearing premise

The entire claim that SDP maintains or improves loss and accuracy rests on the FLOP-matched comparison protocol, which gives SDP roughly $1/C$ times as many update steps as DDP for the same total FLOPs; if equal step count or equal wall-clock time is instead the right basis, the reported loss and accuracy parity would likely not hold.

Editorial extensions

If this is right

  • At 1B LLaMA scale, B-SDP with coverage $C=0.69$ cuts peak GPU memory by 28% while reaching equal or lower validation loss, and all three SDP variants stay within about one point of DDP on five downstream benchmarks.
  • B-SDP composes with FSDP ZeRO-3 and activation checkpointing: the three-way combination at 1B uses 9.7 GB per device, 85% below DDP and below either FSDP or activation checkpointing alone, and B-SDP plus FSDP is the fastest configuration in both intra-node and inter-node profiling.
  • On vision architectures, ResNet-18 retains DDP accuracy at 40% of DDP memory on CIFAR-10, and B-SDP on Swin-Tiny improves CIFAR-100 accuracy by about 2 points while using 32% less memory.
  • At aggressive sparsity ($C=3/8$), forward-masked variants collapse on ResNet/WRN, whereas backward-masked Bb-SDP degrades gracefully and prevents collapse.
  • The per-step communication volume scales with the active fraction $C$, so SDP's benefit widens when interconnect bandwidth is the bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the performance-parity result is built on the FLOP-matched protocol, the practical value of SDP hinges on whether extra iterations can be amortized; a deployment comparing time-to-target rather than total FLOPs could see the memory savings partly converted into a quality gap at equal step count.
  • The moderate-sparsity accuracy gains on CIFAR suggest an implicit regularization or ensembling effect of overlapping fixed subnetworks; a controlled test would hold memory constant and compare SDP at $C=0.75$ against DDP with strengthened dropout or weight decay to see whether the gains are specific to the subnetwork mechanism.
  • The convergence bound ties quality loss to the mask's distance from uniform averaging $\rho^2$, pointing to a mask-design objective the paper does not optimize: choose assignments to minimize that distance subject to the memory budget, then measure whether the validation-loss gap at fixed $C$ shrinks.
  • Since SDP reduces all communication to parameter and gradient traffic, it should be particularly attractive on slow interconnects; a stress test that throttles network bandwidth below the profiled NVLink/InfiniBand regimes would quantify how much of the wall-clock advantage survives under weaker hardware.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Subnetwork Data Parallelism (SDP), a distributed training framework in which each worker trains a fixed subnetwork of the model, defined by structured masks, and overlapping parameters are synchronized by masked averaging. Two masking regimes are studied: forward masking (B-SDP, N-SDP), which removes computation in both forward and backward passes, and backward masking (Bb-SDP), which keeps a full forward pass but applies sparsity only in the backward pass. The authors claim memory and communication reductions proportional to the coverage ratio C, supported by experiments on LLaMA models (134M/500M/1B) pre-trained on FineWeb and on ResNet/Swin models on CIFAR, reporting that SDP reduces peak memory by 28%-60% while maintaining or improving validation accuracy under a FLOP-matched protocol. A convergence theorem for backward masking is stated in Section 3.1.

Significance. Should the empirical claims hold, SDP would be a practical and complementary alternative to data parallelism: it reduces per-device memory across parameters, gradients, optimizer states, and activations simultaneously, and it composes with FSDP and activation checkpointing (Figure 3). The paper's strengths include measured communication volumes and wall-clock times (Table 2), downstream evaluation on five benchmarks (Table 1), a broad architectural sweep (LLaMA, Swin, ResNet, WideResNet), and a candid limitations appendix. The B-SDP results, in particular, are supported by wall-clock measurements consistent with the claimed FLOP-matched budget. However, the convergence theorem is not a valid SGD rate, and the FLOP-matched protocol is not correctly realized for Bb-SDP and N-SDP, so the headline 'maintaining performance under FLOP-matched settings' is only established for B-SDP at present.

major comments (3)
  1. [3.1, Eq. (2)] The convergence bound in Theorem 3.1 is not a valid SGD rate. When the backward mask equals the uniform mask (m = m_uni, so c = n), ρ = 0 and the right-hand side of Eq. (2) reduces to 2(f(θ0)-f*)/(ηT), which tends to 0 as T grows; this is impossible for stochastic gradients, which have a noise floor of σ²/n that does not vanish. The proof in Appendix B (Lemma B.1) bounds E||P x - \bar{x}||², the expected squared deviation between the masked average and the uniform average, and then replaces σ² in the standard SGD bound with this quantity. But the SGD noise term is the variance of the gradient estimator around its mean ∇f, which for a coordinate averaged over c workers is σ²/c (or σ²/n for uniform averaging), not the distance between two averaging operators. A correct bound must retain a σ²/c floor even when the mask is uniform. Furthermore, for constant column coverage c, ρ² = |J|(1/c - 1/n) with |J| the number of coordinates, so the stated bound degrades with model size and is vacuous for large networks. This invalidates the paper's claim of a theoretically grounded backward-masking baseline.
  2. [Section 4 (Experiments)] The FLOP-matched protocol for Bb-SDP is inconsistent with the method's actual per-step compute. Section 4 states that Bb-SDP 'receives an additional 1.5× iterations at the same C to absorb the higher backward cost.' If 'additional' means 1.5× on top of the 1/C scaling used for forward masking, then at C = 0.69 Bb-SDP runs roughly 2.17× DDP's iteration count; if it means a total of 1.5× DDP iterations, it is still above the FLOP-matched value. For a backward-masked step with a full forward pass and masked backward pass, the per-step FLOP ratio to DDP is (1+C)/2 under the equal-cost assumption for forward and backward, and (1+2C)/3 if backward costs twice the forward. FLOP-matching therefore requires 2/(1+C) ≈ 1.18 iterations at C = 0.69 (or ≈ 1.26 under the second assumption), not 1.5. No measured FLOP counts are reported, so the claimed compute equality between Bb-SDP and DDP is unverified; if 1.5× was actually used, the validation-loss parity in Figure 1 and Table 1 for Bb-SDP may reflect additional compute rather than a property of the method.
  3. [Section 4.1 (N-SDP)] For N-SDP, the coverage ratio C is defined as the fraction of active attention heads, but the MLP sublayers in each transformer block are not masked. The fraction of active parameters and the per-step FLOP count are therefore strictly larger than C. The protocol in Section 4 scales training iterations by 1/C, which grants N-SDP more compute than a true FLOP-matched comparison against DDP. For instance, if attention parameters are roughly one-third of the model, then at C = 0.69 the actual active-parameter fraction is near 0.90, and FLOP-matching would permit only about 1.11× DDP's iterations rather than 1.45×. This over-crediting affects the Pareto claim for N-SDP in Figure 1 and the downstream results in Table 1. The manuscript should define C over all parameters (including MLPs) or report the measured active-parameter fraction and per-step FLOPs for every reported configuration.
minor comments (6)
  1. [1 (Introduction)] The contribution bullet advertises 'a spectral-gap condition on the mask graph,' but Theorem 3.1 is expressed through the Frobenius norm ρ and no spectral-gap quantity is defined or analyzed anywhere in the manuscript.
  2. [Figure 1 / Table 1] Figure 1 and Table 1 do not report variance across seeds for the LLaMA experiments, even though Section 4 says results are over multiple random seeds.
  3. [Appendix A] Appendix A contains garbled summation notation: 'Pm j=1' and 'Pn i=1' should be formatted as proper sums; the same typo appears in Section 3.1.
  4. [Table 2] Table 2 has a stray 'torch' in the column header ('vs. DDP torch Step (ms)'); please clean up the table formatting.
  5. [4.1.1 / Figure 3] Figure 3 includes a 'ZeRO-2' label in the bar legend, but the text of Section 4.1.1 only discusses FSDP and activation checkpointing; clarify whether ZeRO-2 is part of the comparison.
  6. [Table 1 caption] The caption describes results as 'zero/few-shot' without specifying the number of shots for each benchmark; please state the exact evaluation protocol.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: SDP results rest on external DDP/FSDP baselines and standard SGD analysis; the FLOP-matched protocol is a correctness risk, not a circular derivation.

full rationale

Walking the derivation chain: the memory-reduction claims are measured against external DDP/FSDP baselines and follow from the chosen coverage ratio C, so they are empirical measurements rather than predictions derived from the paper's own assumptions. The performance claims are validated against DDP and FSDP using validation loss and external downstream benchmarks (lm-evaluation-harness); no fitted parameter is relabeled as a prediction. Theorem 3.1 is a standard L-smooth SGD variance bound (Appendix B) with a mask-distance term rho; it neither assumes the empirical Pareto conclusions nor is used to fit them. The self-citations (WASH [12], PETRA [35], Decoupled Greedy Learning [3]) appear only in related-work and post-hoc attribution contexts and are not load-bearing for the central claims. The only substantive concern is Section 4's FLOP-matching protocol: for Bb-SDP the stated 'additional 1.5x iterations at the same C' is not justified by the (1+C)/2 FLOP ratio, and N-SDP's C counts only attention heads while MLP parameters remain unmasked; since no measured FLOPs are reported, equality of FLOP budgets is an unverified assumption. This is a potential experimental confound, not a circular reduction: the results are not true by construction under a correct protocol, and an incorrect protocol would undermine the FLOP-matched claim rather than tautologically establish it. Appendix I states the actual limitations (scale and architectural coverage) and does not reveal any circular step. No self-definitional, fitted-input-as-prediction, or self-citation-reduction pattern is present, so the paper is self-contained with respect to circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claims rest on a handful of standard smoothness assumptions, a heuristic balanced-mask construction, and architecture-specific masking properties. No new physical or architectural entities are introduced. The coverage ratio C and per-cell learning rate are user-chosen controls that determine the reported tradeoff curves.

free parameters (2)
  • Coverage ratio C = Values from 0.33 to 0.88 across experiments.
    Fraction of blocks or neurons active on each worker; swept to produce the memory-accuracy Pareto plots. The 28-60% memory savings headline depends on choosing C in this range.
  • Learning rate per (method, coverage) cell = Best value chosen from the set 1e-3, 2e-3, 4e-3, 6e-3, 8e-3, 1e-2 for each LLaMA cell.
    Appendix E reports a per-cell LR sweep and selects the best validation loss per cell for the Pareto points, which can inflate apparent gains over a fixed-hyperparameter comparison.
assumptions (5)
  • domain assumption The loss f is L-smooth and worker gradient estimates have bounded variance sigma squared.
    Used in Theorem 3.1 (Section 3.1, Eq. (1)); standard SGD analysis assumption, unverified for transformer or ResNet training.
  • ad hoc to paper Balanced mask construction (Algorithm 1) produces masks with near-uniform column coverage and exact per-worker budget p.
    Appendix A says the procedure 'often satisfies' the balanced property; the coverage ratio C and the theoretical distance rho squared depend on this heuristic.
  • domain assumption Structured masks preserve a full input-to-loss path: dropping a neuron or block consistently removes the corresponding inputs in the next layer.
    Section 3.2 relies on dropout- and stochastic-depth-style consistency (Eq. (3)) so each worker's subnetwork remains end-to-end valid.
  • standard math Residual skip connections let dropped blocks reduce to identity.
    Used in Eq. (4) for B-SDP block masks; holds for ResNet and Transformer blocks with additive residuals.
  • domain assumption Group normalization is used in place of batch normalization so normalization is computed only over active parameters.
    Appendix C changes the architecture relative to standard ResNet; the DDP baseline also uses this normalization, but it is a deliberate design choice rather than the default.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Model Parallelism With Subnetwork Data Parallelism." pith.science (2026). https://pith.science/paper/NMLUEL2B

@misc{pith2026250709029,
  author       = {Pith},
  title        = {Pith review of: Model Parallelism With Subnetwork Data Parallelism},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NMLUEL2B}},
  note         = {Machine review of arXiv:2507.09029}
}
read the original abstract

Pre-training large neural networks at scale imposes heavy memory demands on accelerators and often requires costly communication. We introduce Subnetwork Data Parallelism (SDP), a distributed training framework that partitions a model into structured subnetworks trained across workers without exchanging activations. We study two complementary masking regimes: backward masking, which applies sparsity only in the backward step to retain unbiased gradients, and forward masking, which also removes parameters in the forward pass to deliver stronger efficiency gains while providing additional regularization. We further explore two subnetwork construction strategies: neuron level and block level, applied across both transformers and CNNs. In experiments spanning 1B LLaMA pre-training on FineWeb to ResNet-18 on CIFAR, SDP reduces per device memory usage by 28%-60% while maintaining or improving performance under FLOP-matched settings.

Figures

Figures reproduced from arXiv: 2507.09029 by the authors.

Figure 1
Figure 1. SDP achieves significant memory reduction at both scales. FLOP-matched LLaMA runs. The coverage ratio C is the fraction of active transformer components (layers/attention heads) per worker; DDP has C= 1. The shaded Pareto region contains configurations that strictly dominate DDP in both peak memory and validation loss (lower the better). The best SDP points reduce maximum peak GPU memory by 40% on 500M (B-SDP, C= 0.… view at source ↗
Figure 2
Figure 2. Data Parallelism (DDP) vs. Subnetwork Data Parallelism (SDP). Left: In data parallelism each GPU hosts a full replica, computes all layer gradients {∇L1, ∇L2, ∇L3, ∇L4}, and all-reduces all parameters each step; per-GPU memory is approximately the full model (parameters + gradients + optimizer state + activations). Right: In SDP each GPU trains an end-to-end subnetwork (a subset of layers/neurons) with a local loss … view at source ↗
Figure 3
Figure 3. B-SDP composes with orthogonal memory-saving techniques. Peak GPU memory per device on 4×H100s, in GB with maximum batch size for each model that could fit DDP. We measure B-SDP with C= 0.50 on 500M and C= 0.69 on 1B, the configurations that Pareto-dominate DDP in val. loss ( [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Top-1 accuracy (↑) versus fraction of DDP memory for B-SDP, Bb-SDP, and N-SDP on Swin tiny architecture. All SDP configurations achieve competitive accuracy while using substantially less memory than the DDP baseline, demonstrating a clear efficiency advantage over ful…
Figure 5
Figure 5. Figure 5: Top-1 accuracy (↑) versus fraction of DDP memory for B-SDP, Bb-SDP, and N-SDP on ResNet-18 across CIFAR-10 and CIFAR-100. SDP configurations consistently achieve competitive or higher accuracy while using substantially less memory (60% reduction) than the DDP baseline.…
Figure 6
Figure 6. Figure 6: 134M LLaMA: SDP Pareto-dominates DDP. Each point is a 134M LLaMA trained under a FLOP￾matched protocol; only the coverage ratio C (annotated) varies. The shaded Pareto region marks configurations that strictly dominate DDP: lower peak GPU memory at no worse validation …
Figure 7
Figure 7. Figure 7: Top-1 accuracy (↑) versus fraction of DDP memory for B-SDP, Bb-SDP, and N-SDP on WideResNet￾18 across CIFAR-10 and CIFAR-100. The trends match ResNet-18 in main-text [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 26 canonical work pages

  1. [1]

    O. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, and S. A. et al. Gpt-4 technical report. 2023. URL https: //api.semanticscholar.org/CorpusID:257532815

    work page 2023

  2. [2]

    S. Alam, L. Liu, M. Yan, and M. Zhang. Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction.Advances in neural information processing systems, 35: 29677–29690, 2022

    work page 2022

  3. [3]

    Belilovsky, M

    E. Belilovsky, M. Eickenberg, and E. Oyallon. Decoupled greedy learning of cnns. InInterna- tional Conference on Machine Learning, pages 736–745. PMLR, 2020

    work page 2020

  4. [4]

    Bommasani, D

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  5. [5]

    L. Bottou. Large-scale machine learning with stochastic gradient descent. InProceedings of COMPSTAT’2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pages 177–186. Springer, 2010

    work page 2010

  6. [6]

    Caldas, J

    S. Caldas, J. Koneˇcny, H. B. McMahan, and A. Talwalkar. Expanding the reach of federated learning by reducing client resource requirements.arXiv preprint arXiv:1812.07210, 2018

    arXiv 2018

  7. [7]

    Y . Cho, B. Shin, C. Kang, and C. Yun. Lightweight dataset pruning without full training via example difficulty and prediction uncertainty.arXiv preprint arXiv:2502.06905, 2025

    arXiv 2025

  8. [8]

    E. Diao, J. Ding, and V . Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients.ICLR, 2021

    work page 2021

Show all 51 references
  1. [9]

    Douillard, Q

    A. Douillard, Q. Feng, A. A. Rusu, R. Chhaparia, Y . Donchev, A. Kuncoro, M. Ranzato, A. Szlam, and J. Shen. Diloco: Distributed low-communication training of language models. arXiv preprint arXiv:2311.08105, 2023

    2023 arXiv

  2. [10]

    Douillard, Y

    A. Douillard, Y . Donchev, J. K. Rush, S. Kale, Z. Charles, G. Teston, Z. Garrett, J. Shen, R. McIlroy, D. Lacey, et al. Streaming diloco with overlapping communication. InSecond Conference on Language Modeling, 2025

    2025

  3. [11]

    Fagnou, P

    E. Fagnou, P. Caillon, B. Delattre, and A. Allauzen. Accelerated training through iterative gradient propagation along the residual path.arXiv preprint arXiv:2501.17086, 2025

    2025 arXiv

  4. [12]

    Fournier, A

    L. Fournier, A. Nabli, M. Aminbeidokhti, M. Pedersoli, E. Belilovsky, and E. Oyallon. Wash: Train your ensemble with communication-efficient weight shuffling, then average.arXiv preprint arXiv:2405.17517, 2024

    2024 arXiv

  5. [13]

    L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. McDonell, N. Muennighoff, J. Phang, L. Reynolds, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou. A framework for few-shot language model evaluation, Sept. 2021. URL https://doi.org/10. 5281/ze...

    2021

  6. [14]

    Goyal, P

    P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y . Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in 1 hour.arXiv preprint arXiv:1706.02677, 2017. 10

    2017 arXiv

  7. [15]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, and A. M. et. al. The llama 3 herd of models, 2024. URLhttps://arxiv.org/abs/2407.21783

    2024 arXiv

  8. [16]

    Guliani, L

    D. Guliani, L. Zhou, C. Ryu, T.-J. Yang, H. Zhang, Y . Xiao, F. Beaufays, and G. Motta. Enabling on-device training of speech recognition models with federated dropout. InICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages ...

    2022

  9. [17]

    K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. InProceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015

    2015

  10. [18]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778, 2016

    2016

  11. [19]

    Hoffmann, S

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  12. [20]

    Horvath, S

    S. Horvath, S. Laskaridis, M. Almeida, I. Leontiadis, S. Venieris, and N. Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout.Advances in Neural Information Processing Systems, 34:12876–12889, 2021

    2021

  13. [21]

    Huang, Y

    G. Huang, Y . Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 646–661. Springer, 2016

    2016

  14. [22]

    Huang, Y

    Y . Huang, Y . Cheng, A. Bapna, O. Firat, M. X. Chen, D. Chen, H. Lee, J. Ngiam, Q. V . Le, Y . Wu, and Z. Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism, 2019

    2019

  15. [23]

    Jolicoeur-Martineau, E

    A. Jolicoeur-Martineau, E. Gervais, K. Fatras, Y . Zhang, and S. Lacoste-Julien. Population parameter averaging (papa).arXiv preprint arXiv:2304.03094, 2023

    2023 arXiv

  16. [24]

    Kirillov, E

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Dollár, and R. Girshick. Segment anything.arXiv:2304.02643, 2023

    2023 arXiv

  17. [25]

    Koneˇcný et al

    J. Koneˇcný et al. Federated optimization: Distributed machine learning for on-device intelli- gence. InarXiv preprint arXiv:1610.02527, 2016

    2016 arXiv

  18. [26]

    Krizhevsky, G

    A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  19. [27]

    Li et al

    S. Li et al. Pytorch distributed: experiences on accelerating data parallel training.Proceedings of MLSys, 2020

    2020

  20. [28]

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021

    2021

  21. [29]

    Oquab, T

    M. Oquab, T. Darcet, T. Moutakanni, H. V . V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y . Huang, H. Xu, V . Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. Labatut,...

    2023

  22. [30]

    Penedo, H

    G. Penedo, H. Kydlíˇcek, A. Lozhkov, M. Mitchell, C. A. Raffel, L. V on Werra, T. Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale.Advances in Neural Information Processing Systems, 37:30811–30849, 2024

    2024

  23. [31]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In M. Meila and T. Zhang, editors,Proceedings of the 38th Internat...

    2021

  24. [32]

    Rajbhandari, J

    S. Rajbhandari, J. Rasley, O. Ruwase, and Y . He. Zero: Memory optimizations toward training trillion parameter models. InSC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020

    2020

  25. [33]

    Rasley, S

    J. Rasley, S. Rajbhandari, O. Ruwase, and Y . He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3505–3506, 2020

    2020

  26. [34]

    S. J. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Koneˇcný, S. Kumar, and H. B. McMa- han. Adaptive federated optimization. InInternational Conference on Learning Representations,

  27. [35]

    Rivaud, L

    S. Rivaud, L. Fournier, T. Pumir, E. Belilovsky, M. Eickenberg, and E. Oyallon. Petra: Par- allel end-to-end training with reversible architectures.International Conference on Learning Representations, 2024

    2024

  28. [36]

    Ryabinin, T

    M. Ryabinin, T. Dettmers, M. Diskin, and A. Borzunov. Swarm parallelism: Training large models can be surprisingly communication-efficient. InInternational Conference on Machine Learning, pages 29416–29440. PMLR, 2023

    2023

  29. [37]

    Shang, K

    J. Shang, K. Schmeckpeper, B. B. May, M. V . Minniti, T. Kelestemur, D. Watkins, and L. Herlant. Theia: Distilling diverse vision foundation models for robot learning. In8th Annual Conference on Robot Learning, 2024. URLhttps://openreview.net/forum?id=ylZHvlwUcI

    2024

  30. [38]

    Shazeer, Y

    N. Shazeer, Y . Cheng, N. Parmar, D. Tran, A. Vaswani, P. Koanantakool, P. Hawkins, H. Lee, M. Hong, C. Young, et al. Mesh-tensorflow: Deep learning for supercomputers.Advances in neural information processing systems, 31, 2018

    2018

  31. [39]

    S. Shi, X. Chu, K. C. Cheung, and S. See. Understanding top-k sparsification in distributed deep learning.arXiv preprint arXiv:1911.08772, 2019

    1911 arXiv

  32. [40]

    Shoeybi et al

    M. Shoeybi et al. Megatron-lm: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  33. [41]

    Srivastava, G

    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting.The journal of machine learning research, 15 (1):1929–1958, 2014

    1929

  34. [42]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023. URLhttps://arxiv.org/abs/2302.13971

    2023 arXiv

  35. [43]

    J. Wang, Y . Lu, B. Yuan, B. Chen, P. Liang, C. De Sa, C. Re, and C. Zhang. Cocktailsgd: Fine- tuning foundation models over 500mbps networks. InInternational Conference on Machine Learning, pages 36058–36076. PMLR, 2023

    2023

  36. [44]

    Wen, K.-J

    D. Wen, K.-J. Jeon, and K. Huang. Federated dropout—a simple approach for enabling federated learning on resource constrained devices.IEEE wireless communications letters, 11(5):923–927, 2022

    2022

  37. [45]

    Wightman, H

    R. Wightman, H. Touvron, and H. Jégou. Resnet strikes back: An improved training procedure in timm.arXiv preprint arXiv:2110.00476, 2021

    2021 arXiv

  38. [46]

    Xu, C.-Y

    H. Xu, C.-Y . Ho, A. M. Abdelmoniem, A. Dutta, E. Bergou, K. Karatsenidis, M. Canini, and P. Kalnis. Grace: A compressed communication framework for distributed machine learning. InProc. of 41st IEEE Int. Conf. Distributed Computing Systems (ICDCS), 2021

    2021

  39. [47]

    B. Yuan, C. R. Wolfe, C. Dun, Y . Tang, A. Kyrillidis, and C. M. Jermaine. Distributed learning of deep neural networks using independent subnet training.arXiv preprint arXiv:1910.02120, 2019

    1910 arXiv

  40. [48]

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong, et al. A survey of large language models.arXiv preprint arXiv:2303.18223, 2023. URL https://arxiv.org/abs/2303.18223. 12

    2023 arXiv

  41. [49]

    Y . Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277, 2023

    2023 arXiv

  42. [50]

    Zhuang, X

    D. Zhuang, X. Zhang, S. Song, and S. Hooker. Randomness in neural network training: Characterizing the impact of tooling.Proceedings of Machine Learning and Systems, 4:316– 336, 2022. 13 A Balanced Mask Construction for SDP Let n be the number of workers, m the number of compo...

    2022

  43. [2021]

    URLhttps://openreview.net/forum?id=LkFG3lB13U5

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.