Pith. sign in

REVIEW 3 major objections 5 minor 41 references

PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that sparse top-k expert routing lets multiple parties collaboratively fine-tune an MoE LLM with near-centralized accuracy, roughly 70% lower peak GPU memory, and resistance to gradient-inversion attacks.

desk verdict Promising protocol for privacy-preserving MoE fine-tuning, but the 'fully robust' privacy claim doesn't survive contact with learned routing. read the letter →

arxiv 2506.02965 v2 pith:JQFZC4WH submitted 2025-06-03 cs.LG

Ze Yu Zhang , Bolin Ding , Bryan Kian Hsiang Low This is my paper
classification cs.LG
keywords Mixture-of-Expertscollaborativetrainingprivacypreservationmemoryefficiencygradientinversionlargelanguagemodelstop-kroutingfederatedlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes PC-MoE, a protocol that lets several parties jointly fine-tune one sparse Mixture-of-Experts (MoE) large language model without sharing raw data, full gradients, or whole model weights. The central claim is that MoE's top-k sparsity can serve both goals at once: memory is reduced because each party stores only a shard of the expert pool, and privacy is protected because the only tensors that cross party boundaries are sparse activations and activation-sized gradients. Reported experiments across seven LLM benchmarks show the protocol nearly matching, and sometimes beating, a fully centralized baseline in converged accuracy and convergence speed, while cutting peak GPU RAM by roughly 70 percent per party. If this holds, collaborative MoE training could give memory- and data-limited organizations the benefits of a large shared model without the usual accuracy cost of privacy-preserving training.

What carries the argument

The load-bearing mechanism is sparse top-k routing combined with round-robin expert sharding. In each MoE layer the router selects at most k of the m global experts; because experts are distributed uniformly across n parties, a single party hosting one routed expert is the expected case and seeing two or more is $O(k^2/n^2)$. The backward pass retraces the forward path in reverse, so an expert host needs only the gradient with respect to that expert's output and returns only the gradient with respect to its input, never transmitting Jacobians or weights. These ingredients yield the paper's per-step risk bound $R \le \frac{k}{n}\frac{\gamma}{(1-\gamma)^2} K\min(kq, q_{\text{total expert}})$, in which $\gamma$ is the decay factor of the exponential collusion prior, quantifying how memory amortization and privacy arise from the same sparsity.

What would settle it

A concrete test would be to run PC-MoE with $n=8$, $k=2$, and a router trained to send most tokens to one party's experts, then apply gradient inversion to the partial gradients accumulated at that party; if reconstruction rises above the reported ROUGE levels, the owner-agnostic routing assumption fails. Alternatively, recompute the risk bound with coalition sizes drawn uniformly rather than from an exponential prior and check whether the resulting risk is still negligible.

Watch

Extended reading notes

Core claim

PC-MoE is built on the observation that expert layers make up most trainable parameters in an MoE LLM (about 93% in the reported setup) even though each token activates only a small top-k set of experts. The protocol assigns the m experts to n parties in round-robin fashion while keeping each party's backbone, gating network, labels, and dataset local; a party's router may call experts hosted on other machines, and those hosts run the expert's forward and backward passes and return only an activation-sized gradient tensor. Because gradients retrace the sparse forward path, no party ever sees another party's internal weights or full gradient. The paper's risk analysis bounds per-step reconstruction probability by a factor proportional to k/n multiplied by a term from the exponentially decaying collusion prior, and its empirical partial-gradient attacks recover only a few percent of reference unigrams and no bigrams. The central claim is that this arrangement supplies near-centralized performance and convergence with roughly 70% peak GPU RAM reduction, breaking the usual trade-off between privacy and accuracy.

Load-bearing premise

The privacy proof depends on assuming that large colluding coalitions are exponentially unlikely and that the router picks experts without regard to which party owns them; if real adversaries can form large coalitions easily, or routing concentrates traffic on a few parties, the stated per-step risk bound no longer holds.

Editorial extensions

If this is right

  • Parties with small GPUs can co-train an MoE LLM at near-centralized quality: per-party expert memory drops to roughly $k/n$ of the vanilla MoE footprint, and reported peak RAM falls by about 70%.
  • Privacy does not require adding differential-privacy-style noise: under the paper's assumptions with $k=2$, $n=8$, the per-step reconstruction risk is reduced by roughly a factor of four versus fully shared gradients, and empirical attacks recover negligible content.
  • No global gradient averaging or weight broadcast is needed; parties only update their local backbone and the experts they host, coordinated by a lightweight round-robin schedule.
  • The reported ablations indicate the benefits grow with the number of parties: more parties mean lower per-party memory and larger gains in converged accuracy and convergence speed relative to isolated training.
  • Sharing all expert layers is reported to be safe and usually better than skipping some; skipping layers tends to slow convergence, increase variance, and move the result toward the isolated baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The privacy guarantee is conditional on the exponential-decay collusion prior; deployments that cannot vouch for that prior should add traffic monitoring or other safeguards before trusting the stated risk bound.
  • The paper does not analyze inversion of the hidden activations themselves while they travel to remote expert hosts; a natural extension is to test whether the sparse intermediate features alone permit property inference or partial input reconstruction.
  • The protocol is evaluated on fine-tuning tasks; extending it to pre-training would require checking the effect of asynchronous local backbone updates on convergence and expert specialization.
  • A testable extension is to replace the binomial routing model with a measured routing distribution; if learned routers are biased toward particular parties, the expected number of gradients a party sees per layer grows, and the risk bound changes accordingly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes PC-MoE, a decentralized fine-tuning protocol for Mixture-of-Experts (MoE) LLMs in which each participating party retains its own backbone and gating layers, hosts a private shard of the global expert pool, and exchanges only sparse top-k expert activations and gradients during training. The authors claim that this protocol matches the performance and convergence rate of a fully centralized model across seven LLM benchmarks, reduces peak GPU RAM usage by roughly two-thirds per party, and is 'fully robust against reconstruction attacks' under a semi-honest threat model with an exponential-decay collusion prior. The manuscript includes a formal privacy analysis in Appendix B, an empirical privacy evaluation against a partial-gradient attack, and ablations on the number of parties and the number of shared expert layers.

Significance. If the utility and memory results hold, PC-MoE is a practically interesting contribution: it offers a concrete architectural mechanism (MoE sparsity) that simultaneously amortizes memory across parties and limits the information each party must reveal, with experiments that include error bars, multiple tasks, and scalability ablations from two to eight parties. The explicit protocol pseudocode and the attempt to provide a formal privacy bound are valuable steps. However, the headline privacy claim is currently stronger than what is established: the theoretical bound rests on assumptions that are in tension with the actual routing algorithm, and the empirical evaluation covers only a single attack on a few datasets. The utility and memory-efficiency contributions are independently supported by measurements, so the paper's core engineering contribution is credible, but the privacy claims need either substantial additional support or significant qualification.

major comments (3)
  1. [Appendix B, Eq. (B3) and Section 3.3] The per-step risk bound is derived under Assumption A2, which states that the k routed experts are sampled independently of their owners, yielding J ~ Binomial(k, 1/n) for the number of gradient pairs a fixed party observes. This contradicts Algorithm 7, where the gating function selects top-k indices from Softmax(W_i h): routing is a deterministic, content-dependent function and will specialize for domain-specific data. A remote expert host may therefore observe a large fraction of a party's hidden activations and gradients, not k/n, and the paper provides no measurement of routing skew across parties or tasks, nor a worst-case bound that removes A2. Because the abstract's 'fully robust' claim is justified by Eq. (B3), this is a load-bearing gap in the privacy argument.
  2. [Appendix B, Eq. (B3) and Section 3.3] The bound contains the unquantified quantity q = Pr(reconstruction | 1 expert grad) from Assumption A3. The final conclusion R ≤ (k/n)·γ/(1−γ)^2·K·min(kq, q_total) ≪ q_total is not supported by the displayed inequality: for the paper's own example (k=2, n=8, γ=0.5), the multiplicative constant is approximately 0.25, giving R ≤ 0.25·min(2q, q_total), which is not a small absolute risk unless q is known to be tiny. The paper should either provide an empirical or theoretical upper bound on q, or state the result as a relative reduction in risk compared to full-gradient sharing.
  3. [Section 4, 'Empirical validation on privacy guarantee'] The empirical claim of robustness rests on a single attack (the partial-gradient attack of Li et al.) applied to three datasets with ten reconstructions each. The paper does not evaluate against stronger attacks such as DAGER, even though it cites that work, and it does not analyze inversion from the hidden activations that remote expert hosts observe during the forward pass; the formal analysis in Appendix B considers only gradients. The abstract's 'fully robust against reconstruction attacks' is therefore stronger than the evidence. The claim should be qualified to refer to the tested attack and the stated semi-honest threat model.
minor comments (5)
  1. [Appendix B] The display for Pr[J = j] appears twice in the derivation; the duplication should be removed.
  2. [Throughout] The method name is typeset with an irregular space in several places ('PC-M OE'); this should be fixed to 'PC-MoE' consistently.
  3. [Abstract and Section 5] The abstract and conclusion state that the method is 'fully robust against reconstruction attacks,' while Section 4 reports only that 'virtually no meaningful content' is recovered; the language should be made consistent and appropriately qualified.
  4. [Table 2 and Section 4] The average relative total RAM is 33.64%, corresponding to a 66.4% reduction, while the text says 'around 70%' and the abstract says 'near 70%'; consider stating the measured average and describing the per-task range (e.g., up to ~80%) instead.
  5. [Section 3.2] There is a typo in the 'No global broadcast' paragraph: 'entirty' should be 'entirety'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found: the utility and memory claims are external experimental measurements, and the privacy bound is an explicit conditional derivation rather than a self-referential reduction.

full rationale

PC-MoE's central claims are supported independently of any self-citation or fitted-input-as-prediction loop. The performance and memory results (Tables 1 and 2) are measured against centralized and isolated baselines; the 70% RAM reduction is an observed quantity, not derived from the k/n assumption. The privacy analysis is explicitly conditional: Eq. B3 follows from stated assumptions A1-A4, including the exponential-decay collusion prior and the owner-agnostic routing assumption A2. This is a theorem relative to assumptions, not a reduction of the conclusion to the input by construction. The empirical privacy validation uses an external partial-gradient attack (Li et al., 2024) given stronger access than the protocol provides, and the observed failure to reconstruct is independent evidence. The paper contains no load-bearing self-citations: references to prior sparsification defenses and expert-parallel systems are contextual and not used to license the central claims. One caveat belongs to correctness rather than circularity: A2 (owner-agnostic routing) may be violated by the content-dependent top-k router in Algorithm 7, so the unconditional 'fully robust' phrasing may overstate the guarantee; however, the paper does not define or derive the guarantee in a way that makes it true by definition, and the empirical attack results stand independently. Accordingly, no circular step can be exhibited with the required specificity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on a small number of modeling assumptions, the most load-bearing being the exponential collusion prior and the owner-agnostic routing assumption. The free parameter gamma tunes the privacy bound; q is an unquantified probability. No new physical entities are introduced.

free parameters (2)
  • gamma = 0.5 (example), tunable
    Controls the exponential-decay collusion prior (Eq. 1); smaller gamma yields stronger privacy bound; no empirical or theoretical justification is given.
  • q
    Pr(reconstruction | one expert gradient) in Eq. B1; not estimated, yet drives the bound in Eq. B3.
assumptions (4)
  • domain assumption Expert selection is independent of expert ownership (owner-agnostic routing)
    Appendix B, Assumption A2. Required for the binomial calculation of how many selected experts a party sees; in practice routing is learned and can be biased.
  • ad hoc to paper Coalition sizes follow an exponentially decaying prior with tunable gamma
    Eq. 1 and Assumption A4. Introduced for this paper; no empirical support; the privacy bound's strength is controlled by gamma.
  • domain assumption Bounded reconstruction from partial gradients, with unknown q
    Assumption A3, Eq. B1. The whole privacy bound depends on this; q is never estimated.
  • domain assumption Semi-honest threat model, no malicious adversaries
    Section 3.1. A malicious party could deviate from routing or send malformed gradients; the analysis does not cover that.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs." pith.science (2026). https://pith.science/paper/JQFZC4WH

@misc{pith2026250602965,
  author       = {Pith},
  title        = {Pith review of: PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JQFZC4WH}},
  note         = {Machine review of arXiv:2506.02965}
}
read the original abstract

Mixture-of-Experts (MoE) has been gaining popularity due to its successful adaptation to large language models (LLMs). In this work, we introduce Privacy-preserving Collaborative Mixture-of-Experts (PC-MoE), which leverages the sparsity of the MoE architecture for memory-efficient decentralized collaborative LLM training, enabling multiple parties with limited GPU-memory and data resources to collectively train more capable LLMs than they could achieve individually. At the same time, this approach protects training data privacy of each participant by keeping training data, as well as parts of the forward pass signal and gradients locally within each party. By design, PC-MoE synergistically combines the strengths of distributed computation with strong confidentiality assurances. Unlike most privacy-preserving schemes, which pay for confidentiality with lower task accuracy, our framework breaks that trade-off: across seven popular LLM benchmarks, it almost matches (and sometimes exceeds) the performance and convergence rate of a fully centralized model, enjoys near 70% peak GPU RAM reduction, while being fully robust against reconstruction attacks.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 20 canonical work pages

  1. [1]

    https: //arxiv.org/abs/2209.01188

    Borzunov, A., Baranchuk, D., Dettmers, T., Ryabinin, M., Belkada, Y ., Chumachenko, A., Raffel, C.: Petals: Collaborative Inference and Fine-Tuning of Large Models. https: //arxiv.org/abs/2209.01188. arXiv:2209.01188 (2022)

    arXiv 2022

  2. [2]

    In: Advances in Neural Information Processing Systems, pp

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A.,et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems, pp. 1877–1901 (2020)

    work page 2020

  3. [3]

    https://arxiv.org/abs/1803.05457

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., Tafjord, O.: Think You Have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. https://arxiv.org/abs/1803.05457. arXiv:1803.05457 (2018)

    arXiv 2018

  4. [4]

    Improving Robustness to Model Inversion Attacks via Sparse Coding Architectures

    Dibbo, S.V ., Breuer, A., Moore, J., Teti, M.: Improving Robustness to Model Inversion Attacks via Sparse Coding Architectures. https://arxiv.org/abs/2403.14772. arXiv:2403.14772 (2024)

    work page Pith review arXiv 2024

  5. [5]

    https://arxiv.org/abs/2311.08105

    Douillard, A., Feng, Q., Rusu, A.A., Chhaparia, R., Donchev, Y ., Kuncoro, A., Ranzato, M., Szlam, A., Shen, J.: DiLoCo: Distributed Low-Communication Training of Language Models. https://arxiv.org/abs/2311.08105. arXiv:2311.08105 (2023)

    arXiv 2023

  6. [6]

    In: International Conference on Machine Learning, pp

    Du, N., Huang, Y ., Dai, A.M., Tong, S., Lepikhin, D., Xu, Y ., Krikun, M., Zhou, Y ., Yu, A.W., Firat, O.,et al.: Glam: Efficient scaling of language models with mixture-of-experts. In: International Conference on Machine Learning, pp. 5547–5569 (2022). PMLR

    work page 2022

  7. [7]

    Enhancing Privacy against Inversion Attacks in Federated Learning by using Mixing Gradients Strategies

    Eloul, S., Silavong, F., Kamthe, S., Georgiadis, A., Moran, S.J.: Enhancing Privacy against Inversion Attacks in Federated Learning by Using Mixing Gradients Strategies. https://arxiv.org/abs/2204.12495. arXiv:2204.12495 (2022) 17

    work page Pith review arXiv 2022

  8. [8]

    Fedus, W., Zoph, B., Shazeer, N.: Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research23(120), 1–39 (2022)

    work page 2022

Show all 41 references
  1. [9]

    16937–16947 (2020)

    Geiping, J., Bauermeister, H., Dröge, H., Moeller, M.: Inverting gradients—how easy is it to break privacy in federated learning? In: Advances in Neural Information Processing Systems, pp. 16937–16947 (2020)

    2020

  2. [10]

    Gauthier, F., Gogineni, V .C., Werner, S., Huang, Y .-F., Kuh, A.: Personalized graph federated learning with differential privacy.IEEE Transactions on Signal and Information Processing over Networks 9(3), 736–749 (2023) https://doi.org/10.1109/TSIPN.2023. 3325963

    2023 doi

  3. [11]

    arXiv:2406.04127 (2024)

    Gema, A.P., Leang, J.O.J., Hong, G., Devoto, A., Mancino, A.C.M., Saxena, R., et al.: Are We Done with MMLU? https://arxiv.org/abs/2406.04127. arXiv:2406.04127 (2024)

    2024 arXiv

  4. [12]

    https://arxiv.org/abs/2203.15556

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., et al.: Training Compute-Optimal Large Language Models. https://arxiv.org/abs/2203.15556. arXiv:2203.15556 (2022)

    2022 arXiv

  5. [13]

    In: Proceedings of Machine Learning and Systems, vol

    Hwang, C., Cui, W., Xiong, Y ., Yang, Z., Liu, Z., Hu, H., Wang, Z., Salas, R., Jose, J., Ram, P., et al.: Tutel: Adaptive mixture-of-experts at scale. In: Proceedings of Machine Learning and Systems, vol. 5, pp. 269–287 (2023)

    2023

  6. [14]

    https://arxiv.org/abs/2407.04153

    He, X.O.: Mixture of a Million Experts. https://arxiv.org/abs/2407.04153. arXiv:2407.04153 (2024)

    2024 arXiv

  7. [15]

    In: Advances in Neural Information Processing Systems, pp

    Huang, Y ., Gupta, S., Song, Z., Li, K., Arora, S.: Evaluating gradient inversion attacks and defenses in federated learning. In: Advances in Neural Information Processing Systems, pp. 7232–7241 (2021)

    2021

  8. [16]

    Hatamizadeh, A., Yin, H., Molchanov, P., Myronenko, A., Li, W., Dogra, P., Feng, A., Flores, M.G., Kautz, J., Xu, D., Roth, H.R.: Do gradient inversion attacks make federated learning unsafe? IEEE Transactions on Medical Imaging 42(7), 2044–2056 (2023) https://doi.org/10.1109/...

    2023

  9. [17]

    Applied Sciences 11(14), 6421 (2021) https://doi.org/10.3390/app11146421

    Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., Szolovits, P.: What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11(14), 6421 (2021) https://doi.org/10.3390/app11146421

    2021 doi

  10. [18]

    In: Advances in Neural Information Processing Systems, pp

    Kojima, T., Gu, S.S., Reid, M., Matsuo, Y ., Iwasawa, Y .: Large language models are zero- shot reasoners. In: Advances in Neural Information Processing Systems, pp. 22199–22213 (2022)

    2022

  11. [19]

    https://arxiv.org/abs/2001.08361

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., et al.: Scaling Laws for Neural Language Models. https://arxiv.org/abs/2001.08361. arXiv:2001.08361 (2020) 18

    2020 arXiv

  12. [20]

    https://arxiv.org/ abs/2006.16668

    Lepikhin, D., Lee, H., Xu, Y ., Chen, D., Firat, O., Huang, Y ., et al.: GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. https://arxiv.org/ abs/2006.16668. arXiv:2006.16668 (2020)

    2020 arXiv

  13. [21]

    https://arxiv.org/abs/2406.00999

    Li, W., Xu, Q., Dras, M.: Seeing the Forest through the Trees: Data Leakage from Partial Transformer Gradients. https://arxiv.org/abs/2406.00999. arXiv:2406.00999 (2024)

    2024 arXiv

  14. [22]

    https://arxiv.org/abs/1809.02789

    Mihaylov, T., Clark, P., Khot, T., Sabharwal, A.: Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. https://arxiv.org/abs/1809.02789. arXiv:1809.02789 (2018)

    2018 arXiv

  15. [23]

    In: Artificial Intelligence and Statistics, pp

    McMahan, B., Moore, E., Ramage, D., Hampson, S., Arcas, B.A.: Communication- efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282 (2017). PMLR

    2017

  16. [24]

    In: Advances in Neural Information Processing Systems, pp

    Petrov, I., Dimitrov, D.I., Baader, M., Müller, M., Vechev, M.: DAGER: Exact gradient inversion for large language models. In: Advances in Neural Information Processing Systems, pp. 87801–87830 (2024)

    2024

  17. [25]

    https://arxiv.org/abs/2104.10350

    Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., et al.: Carbon Emissions and Large Neural Network Training. https://arxiv.org/abs/2104.10350. arXiv:2104.10350 (2021)

    2021 arXiv

  18. [26]

    In: IEEE INFOCOM 2024-IEEE Conference on Computer Communications, pp

    Pan, X., Lin, W., Shi, S., Chu, X., Sun, W., Li, B.: Parm: Efficient training of large sparsely-activated models with dedicated schedules. In: IEEE INFOCOM 2024-IEEE Conference on Computer Communications, pp. 1880–1889 (2024). IEEE

    2024

  19. [27]

    In: International Conference on Machine Learning, pp

    Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Aminabadi, R.Y ., Awan, A.A., Rasley, J., He, Y .: Deepspeed-moe: Advancing mixture-of-experts inference and training to power next- generation ai scale. In: International Conference on Machine Learning, pp. 18332–18346 (2022). PMLR

    2022

  20. [28]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Sarlin, P.-E., DeTone, D., Malisiewicz, T., Rabinovich, A.: Superglue: Learning feature matching with graph neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4938–4947 (2020)

    2020

  21. [29]

    https://arxiv.org/abs/1701.06538

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. https://arxiv.org/abs/1701.06538. arXiv:1701.06538 (2017)

    2017 arXiv

  22. [30]

    https: //arxiv.org/abs/1909.08053

    Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., Catanzaro, B.: Megatron- LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. https: //arxiv.org/abs/1909.08053. arXiv:1909.08053 (2019)

    2019 arXiv

  23. [31]

    https: //arxiv.org/abs/2210.09261

    Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y ., Chung, H.W., et al.: Chal- lenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. https: //arxiv.org/abs/2210.09261. arXiv:2210.09261 (2022) 19

    2022 arXiv

  24. [32]

    In: 2017 IEEE Symposium on Security and Privacy (SP), pp

    Shokri, R., Stronati, M., Song, C., Shmatikov, V .: Membership inference attacks against machine learning models. In: 2017 IEEE Symposium on Security and Privacy (SP), pp. 3–18 (2017). IEEE

    2017

  25. [33]

    https://arxiv.org/abs/2007.05558

    Thompson, N.C., Greenewald, K., Lee, K., Manso, G.F.: The Computational Limits of Deep Learning. https://arxiv.org/abs/2007.05558. arXiv:2007.05558 (2020)

    2020 arXiv

  26. [34]

    IEEE Transactions on Information Forensics and Security 15(8), 3454–3469 (2020) https://doi.org/10.1109/TIFS.2020.2988575

    Wei, K., Li, J., Ding, M., Ma, C., Yang, H.H., Farokhi, F., Jin, S., Quek, T.Q.S., Poor, H.V .: Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security 15(8), 3454–3469 (2020) https://doi.org/10...

    2020

  27. [35]

    In: 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS), pp

    Wei, W., Liu, L., Wu, Y ., Su, G., Iyengar, A.: Gradient-leakage resilient federated learning. In: 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS), pp. 797–807 (2021). IEEE

    2021

  28. [36]

    https://arxiv.org/abs/2206.07682

    Wei, J., Tay, Y ., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., et al.: Emergent Abilities of Large Language Models. https://arxiv.org/abs/2206.07682. arXiv:2206.07682 (2022)

    2022 arXiv

  29. [37]

    In: 2024 IEEE Inter- national Parallel and Distributed Processing Symposium (IPDPS), pp

    Yao, J., Anthony, Q., Shafi, A., Subramoni, H., Panda, D.K.D.: Exploiting inter-layer expert affinity for accelerating mixture-of-experts model inference. In: 2024 IEEE Inter- national Parallel and Distributed Processing Symposium (IPDPS), pp. 915–925 (2024). IEEE

    2024

  30. [38]

    https: //arxiv.org/abs/2304.06364

    Zhong, W., Cui, R., Guo, Y ., Liang, Y ., Lu, S., Wang, Y ., Saied, A., Chen, W., Duan, N.: AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models. https: //arxiv.org/abs/2304.06364. arXiv:2304.06364 (2023)

    2023 arXiv

  31. [39]

    https://arxiv.org/abs/2505.09343

    Zhao, C., Deng, C., Ruan, C., Dai, D., Gao, H., Li, J., Zhang, L., Huang, P., Zhou, S., Ma, S., et al.: Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures. https://arxiv.org/abs/2505.09343. arXiv:2505.09343 (2025)

    2025

  32. [40]

    https://arxiv.org/abs/2206.07284

    Zhang, R., Guo, S., Wang, J., Xie, X., Tao, D.: A Survey on Gradient Inversion: Attacks, Defenses and Future Directions. https://arxiv.org/abs/2206.07284. arXiv:2206.07284 (2022)

    2022 arXiv

  33. [41]

    In: Advances in Neural Information Processing Systems (2019) 20

    Zhu, L., Liu, Z., Han, S.: Deep leakage from gradients. In: Advances in Neural Information Processing Systems (2019) 20

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.