Pith. sign in

REVIEW 4 major objections 6 minor 41 references

Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper derives a joint scaling law for dense and mixture-of-experts language models and argues that MoE can be more memory-efficient than dense models under fixed compute and memory budgets.

desk verdict Useful joint MoE scaling law and a real memory-efficiency result, but the quantitative extrapolations need better validation and uncertainty before they are cited. read the letter →

arxiv 2502.05172 v2 pith:QK2Y7X4S submitted 2025-02-07 cs.LG cs.AIcs.CL

Jan Ludziejewski , Maciej Pióro , Jakub Krajewski , Maciej Stefaniak , Micha{l} Krutul , Jan Ma{l}aśnicki , Marek Cygan , Piotr Sankowski
show 3 more authors
Kamil Adamczewski Piotr Mi{l}oś Sebastian Jaszczur
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords mixtureofexpertsscalinglawslanguagemodelscompute-optimaltrainingmemoryefficiencyexpertcountlearningratetransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish a single scaling law that covers both dense and mixture-of-experts (MoE) language models, so that model size, dataset size, and the number of experts can be chosen under explicit compute and memory budgets. It claims that once the number of experts enters the law correctly, MoE models can be more memory-efficient than dense models: at the same training budget and the same loss, an MoE can fit in less memory. The argument rests on a fitted power-law surface, validated by more than 280 training runs up to 2.7B active and 5B total parameters, and on direct 1.1B-parameter comparisons where MoE matches or beats overtrained dense models. A practitioner would care because most deployment settings are memory-bound, and this paper gives a recipe for when to prefer MoE and how to set the number of experts.

What carries the argument

The load-bearing object is Equation (6), a Chinchilla-style power law whose coefficients and exponents depend on the number of experts through power laws and logarithms: $m(E)=a\hat{E}^{\delta}$, $n(E)=b\hat{E}^{\omega}$, $\mu(E)=\alpha+\gamma\ln\hat{E}$, $\nu(E)=\beta+\zeta\ln\hat{E}$, with the monotone transform $1/\hat{E}=1/(E-1)+(1/E_{\mathrm{start}}-1/E_{\mathrm{max}})^{-1}+1/E_{\mathrm{max}}$ modeling startup overhead and saturation in expert count. This single surface lets the authors solve the constrained optimization problems for compute-only, memory-only, total-memory with KV cache, and joint training-plus-inference budgets, producing tables of optimal $E$ and the rule of thumb that $E\le 8$ experts with $E$ times more tokens can beat a compute-optimal dense model of the same parameter count.

What would settle it

Train a fixed-memory MoE at large scale with an expert count and token-to-parameter ratio deliberately outside the fitted grid, for example a 1.1B-total-parameter $E=8$ model trained well beyond the paper's covered ratio of about 414, and compare its final loss to the predicted memory-optimal curve; a systematic upward bend in the loss would overturn the large-scale memory-efficiency recommendation.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the loss of a Switch-MoE Transformer is described by the joint law $L(N_{\mathrm{act}}, D, \hat{E})=a\hat{E}^{\delta}N_{\mathrm{act}}^{\alpha+\gamma\ln\hat{E}}+b\hat{E}^{\omega}D^{\beta+\zeta\ln\hat{E}}+c$, with $\hat{E}$ a monotone transform of the expert count $E$, and that optimizing this law under $6N_{\mathrm{act}}D=F$ together with a memory constraint makes MoE the optimal choice over broad budget ranges. Because the exponent on tokens grows with $E$, more experts shift the compute-optimal configuration toward more tokens and fewer active parameters. The fitted coefficients predict that an MoE with the same loss and training budget as a dense model can use less memory, and the paper validates this with matched 1.1B-parameter models where $E=2$ and $E=4$ attain lower loss than an overtrained dense model. It also derives a learning-rate scaling rule $\mathrm{LR}(N_{\mathrm{act}\setminus e}, E)=\exp(8.39-0.81\ln N_{\mathrm{act}\setminus e}-0.25\ln E)$, which keeps dense-versus-MoE comparisons unbiased.

Load-bearing premise

The central load-bearing premise is that the fitted power-law loss surface keeps holding far beyond the training range, including budgets up to $5\times10^{22}$ FLOPs, up to 32 experts, and token-to-parameter ratios above about 414.

Editorial extensions

If this is right

  • For any fixed compute budget, the optimal number of active parameters falls and the optimal number of training tokens rises as the number of experts increases.
  • Under a fixed total-memory constraint, an MoE with the right number of experts can reach a lower loss than a dense model trained with the same FLOPs, as demonstrated at the 1.1B-parameter scale.
  • Including the KV cache and inference-time compute makes MoE look even better, because fewer active parameters cut both memory and per-token FLOPs.
  • As a conservative rule, up to eight experts with about $E$ times more training tokens can outperform a compute-optimal dense model at the same memory footprint.
  • MoE learning rates should be lowered as the number of experts grows, following the fitted $\mathrm{LR}(N_{\mathrm{act}\setminus e}, E)$ rule, so that dense-versus-MoE comparisons are not biased by mistuned optimizers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the law transfers to fine-grained or dropless MoE variants, which the paper names as future work, the memory-efficiency advantage would likely strengthen because finer expert granularity improves the compute-to-memory trade-off.
  • The rule of thumb implies that data demand grows roughly linearly with $E$, so dataset availability, not hardware, may become the real ceiling on the optimal number of experts outside web-scale NLP.
  • The learning-rate result suggests that earlier dense-versus-MoE comparisons using a single learning rate for both may have systematically understated MoE; re-tuning with an $E$-dependent learning rate could revise published efficiency rankings.
  • The optimal-expert tables are budget-dependent: at the smallest memory budget considered, dense models remain optimal even at very large FLOPs, so the memory-efficiency claim is not universal but holds in specific budget regimes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a joint scaling law for dense and mixture-of-experts transformers, L(Nact, D, Ehat) = a Ehat^delta Nact^(alpha+gamma ln Ehat) + b Ehat^omega D^(beta+zeta ln Ehat) + c, fitted to over 280 runs with up to 2.7B active and 5B total parameters. It then uses the fitted surface to derive compute-optimal, memory-optimal, and inference-optimal configurations, concluding that MoE models can be more memory-efficient than dense models, and validates this with 1.1B-parameter dense and MoE comparisons. The paper also derives a learning-rate scaling rule for MoE and reports a good empirical fit with RMSE_v around 0.0039.

Significance. If the fitted scaling law is correct and the extrapolations are justified, the paper provides a practically useful framework for choosing MoE configurations under memory constraints, and the finding that MoE can be memory-optimal is an important counter to conventional wisdom. The experiment scale (over 280 runs, with a detailed listing in Appendix E), the learning-rate scaling ablation in Section 5.1.2, and the explicit acknowledgment of extrapolation limits in Appendix C are notable strengths. However, the headline quantitative recommendations depend on extrapolation well beyond the fitted data range, and the validation procedure has circularity issues, so the current evidence is not fully commensurate with the strength of the claims.

major comments (4)
  1. [Section 4.2 and Appendix E] The 1.1B dense and E=2/4 models used to validate the memory-efficiency claim are listed in Appendix E, which Section 5.2 describes as the set of training runs used to fit Eq. (6). This means the 'validation' is not independent of the fit; the comparison in Figure 1(b) could reflect the fitted surface rather than confirm it. Please hold out these runs from the fitting procedure, or provide a fresh set of validation experiments, before claiming empirical confirmation.
  2. [Section 5.2 and Appendix B] The reported held-out RMSE_v=0.0039 is computed on 'a held-out set of our 30 runs with the lowest loss,' while Appendix B states that the coefficients were selected by minimizing the sum of RMSE on the training set and 'a held-out extrapolation validation set.' If these are the same set, RMSE_v is a model-selection score rather than an unbiased generalization estimate, and the extrapolation-quality claims in Figure 5 are weaker than stated. Please clarify the relationship between the two sets; if they coincide, add a genuinely unseen validation set.
  3. [Section 4.5, Table 2, Figure 3, and Appendix C] The optimal-E tables and Figure 3 evaluate Eq. (6) at token-to-parameter ratios and active-parameter counts far outside the fitted grid. Appendix E shows the largest D/N ratios in the fitted data are roughly 73 (E=4), 68 (E=8), and 50 (E=16/32), yet Appendix C itself flags D/N about 414 as outside the covered range; the Rule of Thumb for E=8 with 1.1B total parameters and 8x tokens implies D/N well above 300, and Table 2/Figure 3 use E=16/32 at 5e22 FLOPs. The recommendations depend on the interaction exponents gamma and zeta in Eq. (6), which are not constrained by data in the high-D/N regime. Please add validation runs in the extrapolated regime or restrict the quantitative recommendations to the covered range; the Section 6 caveat about Chinchilla-form limitations at extreme token-to-parameter ratios reinforces this concern.
  4. [Section 4.2, Finding 2] The statement that 'increasing the number of experts always improves performance' is presented as a finding, but within the paper it is a consequence of the fitted surface rather than an empirical result; the direct experiments cover only E=2 and E=4. The paper should either present this as a prediction of the fitted law or provide experimental support for larger E.
minor comments (6)
  1. [Eq. (4)] The definition of Ehat appears to contain a typo: the middle term is typeset as (1/Estart - 1/Emax)^(-1) in a way that makes the equation dimensionally inconsistent as written; please fix and clarify the intended transformation.
  2. [Section 5.1.2] The learning-rate scaling fit uses E=1 and E=8 for training and E=4 and E=32 for validation, but it is not stated whether these grid runs are part of the Appendix E set or separate; please clarify to avoid ambiguity.
  3. [Figure 5 and Section 5.2] Figure 5(a) reports a maximum absolute error of 0.018 while Section 5.2 reports RMSE_v=0.0039; please report the RMSE for the same set to avoid an apparent inconsistency.
  4. [Section 4.5] The Rule of Thumb phrase 'trained on E times more tokens' is immediately qualified by the statement that compute-matched MoE generally uses less than E times more tokens; please rephrase to avoid confusing readers.
  5. [Page 1 footnote] The footnote states that code and checkpoints will be open-sourced; please include a formal reproducibility statement in the final version.
  6. [Table 3 and Appendix D] Minor typos: 'joined formula' in the Table 3 caption should be 'joint formula', and the figure captions in Appendix D use 'exp_rate' where a different label would be clearer.

Circularity Check

2 steps flagged · score 6.0 of 10

The 1.1B validation runs in Section 4.2 appear in the same Appendix E inventory used to fit Eq. 6, so the headline memory-efficiency validation is not an out-of-sample test; the large-scale optimal-E tables are extrapolations of that same fitted surface.

  1. fitted input called prediction [Section 4.2, Section 5.2, Appendix B, Appendix E]
    "We validate this claim by training a 1.1B dense model and a model size and FLOP matched E = {2, 4} counterparts (Figure 1). ... we use the LBFGS algorithm ... to fit the coefficients of Equation 6 ... over the set of our training runs described in the table in Appendix E. ... 1.1B 26 26 1664 1.1B 1 14.0B, 12.0B, 10.0B, 80.0B, 64.0B, 48.0B, 32.0B ... 1.1B 22 22 1408 709M 2 21.8B, 18.7B, 15.6B, 35.3B ... 1.1B 18 18 1152 426M 4 31.0B, 25.9B, 20.7B, 10.4B, 5.2B, 2.6B, 1.3B"

    The 1.1B dense and MoE runs presented as validation in Section 4.2 are listed in Appendix E, the same experiment inventory used as the training set for fitting Equation 6. Their lower loss is therefore already encoded in the fitted coefficients; reporting them as confirmation of the scaling-law prediction is not an out-of-sample test. The comparison may still be a valid empirical observation about those specific runs, but as presented it does not independently verify the fitted surface or the derived optimal-E recommendations.

  2. other [Section 5.2 and Appendix B]
    "We observe a good fit with RMSEv = 0.0039 on a held-out set of our 30 runs with the lowest loss, and RMSEt = 0.0062 on the training dataset. ... The selected coefficients were those with the lowest score, defined as the sum of RMSE on the training and a held-out extrapolation validation set."

    The reported validation RMSE is not a clean generalization estimate because the same held-out set was used to select among candidate coefficients. The low RMSEv is partly a product of selection on that set, so citing it as evidence that the joint formula extrapolates reliably is weaker than stated. This is a validation-contamination issue rather than a full reduction of the prediction to its inputs, but it compounds the reuse of the 1.1B runs in the training set.

full rationale

The core scaling law, Equation 6, is an explicitly postulated fitted ansatz; deriving optimal-E tables by optimizing that fitted surface is standard scaling-law practice and is not circular in itself. The self-citations to Clark et al. (2022) and Ludziejewski et al. (2024) motivate the functional form, but the coefficients are fit to the authors' own 280-run dataset, so the self-citations are not the load-bearing evidence for the quantitative claims. The principal circular step is the Section 4.2 validation: the 1.1B dense and MoE runs that are said to validate the memory-efficiency claim appear in the same Appendix E table used as the training set for Equation 6, so their agreement with the fitted surface is expected rather than independently confirmed. In addition, the reported held-out RMSEv is weakened because the held-out set participated in coefficient selection. The extrapolation to larger budgets, E=16/32, and token-to-parameter ratios near or above the fitted range is a genuine correctness risk, but that is a modeling-risk concern, not circularity. Overall, the central validation reduces to in-sample agreement, giving partial circularity and a score of 6.

Assumptions & free parameters 14 free parameters · 7 assumptions · 0 invented entities

Almost all of the paper's output is purchased by fitting the coefficients in Table 3 to 280 runs, plus the separate learning-rate coefficients in Equation 8. There are no invented physical or architectural entities; the new objects are the fitted coefficients and the empirical formula. The central caveat is that any 'prediction' at budgets above the training range is an extrapolation of a fitted curve, not an independent test.

free parameters (14)
  • active-parameter prefactor a = 35.91
    Fitted scale for the active-parameter term in Equation 6; absorbs the overall loss scale.
  • active-parameter exponent alpha = -0.1889
    Fitted power-law exponent for Nact in the joint scaling law.
  • E-power exponent delta = -0.2285
    Fitted exponent controlling how E scales the active-parameter term.
  • log-E interaction gamma = 0.0098
    Fitted interaction making the Nact exponent depend on ln(E).
  • dataset prefactor b = 35.98
    Fitted scale for the dataset-size term in Equation 6.
  • dataset exponent beta = -0.1775
    Fitted power-law exponent for D in the joint scaling law.
  • E-power exponent omega = 0.5529
    Fitted exponent controlling how E scales the dataset term.
  • log-E interaction zeta = -0.0259
    Fitted interaction making the D exponent depend on ln(E).
  • irreducible loss c = 1.3637
    Fitted floor of the loss, assumed independent of E.
  • Estart = 2.0732
    Fitted parameter in the E-transform of Equation 4; controls small-E overhead.
  • Emax = 290.4521
    Fitted parameter in the E-transform of Equation 4; controls saturation at large E.
  • LR scaling intercept = 8.39
    Fitted intercept in Equation 8 for the peak learning rate.
  • LR scaling Nact exponent = -0.81
    Fitted exponent on active non-embedding parameters in Equation 8.
  • LR scaling E exponent = -0.25
    Fitted exponent on number of experts in Equation 8; used to set learning rates for all training runs.
assumptions (7)
  • domain assumption The Chinchilla law L(N,D)=m N^-mu + n D^-nu + c holds for each fixed number of experts.
    Equation 5 in Section 3 postulates this form without derivation; all later structure builds on it.
  • domain assumption The irreducible loss c is independent of architecture and dataset size.
    Stated under 'Constant Factor' in Section 3; if c varied with E, the joint fit would be biased.
  • ad hoc to paper Power-law dependence on E and logarithmic interactions are the correct functional form.
    Section 3 chooses m(E)=aE^delta, n(E)=bE^omega, and exponents linear in ln(E) based on observations from related work, not from a derivation.
  • domain assumption The Ehat transformation from Clark et al. (2022) correctly captures small-E overhead and large-E saturation.
    Equation 4 is imported from Clark et al. and its parameters Estart and Emax are fitted to the authors' own data.
  • domain assumption FLOPs and memory estimates are accurate: training cost 6 Nact D, inference cost 2 Nact D, and routing cost is negligible.
    Appendix A.2; if routing, communication, or implementation overheads are significant, the compute and memory comparisons would shift.
  • ad hoc to paper The learning-rate scaling law of Equation 8 gives near-optimal learning rates for every configuration.
    All 280 runs use learning rates from this fitted formula; if it mispredicts the optimum for some E or size, loss comparisons across configurations are confounded.
  • domain assumption Capacity-factor MoE with auxiliary losses and token dropping does not invalidate cross-E comparisons.
    Section 5.1 uses Switch layers with z-loss and load-balancing loss; these auxiliary terms can affect small-E and large-E models differently.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient." pith.science (2026). https://pith.science/paper/QK2Y7X4S

@misc{pith2026250205172,
  author       = {Pith},
  title        = {Pith review of: Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QK2Y7X4S}},
  note         = {Machine review of arXiv:2502.05172}
}
read the original abstract

Mixture of Experts (MoE) architectures have significantly increased computational efficiency in both research and real-world applications of large-scale machine learning models. However, their scalability and efficiency under memory constraints remain relatively underexplored. In this work, we present joint scaling laws for dense and MoE models, incorporating key factors such as the number of active parameters, dataset size, and the number of experts. Our findings provide a principled framework for selecting the optimal MoE configuration under fixed memory and compute budgets. Surprisingly, we show that MoE models can be more memory-efficient than dense models, contradicting conventional wisdom. To derive and validate the theoretical predictions of our scaling laws, we conduct over 280 experiments with up to 2.7B active parameters and up to 5B total parameters. These results offer actionable insights for designing and deploying MoE models in practical large-scale training scenarios.

Figures

Figures reproduced from arXiv: 2502.05172 by the authors.

Figure 1
Figure 1. (a) The loss of memory-constrained models predicted using our scaling law under a fixed training budget of 1022 FLOPs. Each curve represents a different number of experts. The lines are truncated at compute-optimal points since undertrained models are both larger and worse in terms of loss, thus pointless in a memory-constrained scenario. Shaded areas indicate the memory-optimal number of experts for the correspondi… view at source ↗
Figure 2
Figure 2. (a) IsoFLOP profiles for selected training budgets, with compute-optimal points marked for each curve. (b) FLOP savings from switching from a compute-optimal dense model to a compute-optimal MoE. For instance, 40% savings at 1e20 FLOPs mean that an MoE matching the performance of a compute-optimal dense model trained with 1e20 FLOPs can be trained with just 6e19 FLOPs (60% of the dense’s budget). The advantage of us… view at source ↗
Figure 3
Figure 3. Predicted loss for various numbers of experts at a FLOPs budget F = 5 × 1022. The x-axis represents the size of the model in terms of the number of parameters (a) or the total memory budget for both model parameters and KV cache for 8192 tokens (b, c). Shaded areas indicate the optimal number of experts for the corresponding parameter or memory budget. (c) In addition to the KV cache, the inference cost on 100B toke… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Investigation of the optimal number of experts for three different model sizes: 2B, 5B, and 10B; and in three different scenarios from left to right: simply measuring the model size, including the size of a KV-cache with 32k tokens, and including the inference cost of …
Figure 5
Figure 5. Figure 5: (a) Quality of the fit. The maximum absolute error on the held-out extrapolation set is 0.018. (b) Predicted loss compared to observed loss for E = 1. (c) Predicted loss (dashed line) compared to observed loss for E = 4. We can see that on the training dataset, the err…
Figure 6
Figure 6. Figure 6: Comparison between compute- and memory-matched models with different values of E. The corresponding total memory constraint for MoE models is derived from the compute-optimal model size for the dense model. Due to the nature of this constraint, we do not consider highe…
Figure 7
Figure 7. Figure 7: Visualization of the fit (E ∈ {1, 8}) of our LR scaling rule, interpolation (E = 4) and extrapolation (E = 32). 2 7 2 8 2 9 2 10 2 11 Learning Rate 3.2 × 10 0 3.4 × 10 0 3.6 × 10 0 3.8 × 10 0 4 × 10 0 Loss Loss vs Learning Rate (exp_rate=1) D: 603M Nact e: 30M D: 1.1B …
Figure 8
Figure 8. Figure 8: Ablation for the LR scaling rule fit without considering the number of experts E. While performance on the training set (E ∈ {1, 8}) looks acceptable, the extrapolation at E = 32 is clearly suboptimal, validating the need for considering E. 16 [PITH_FULL_IMAGE:figures…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 17 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Abnar, S., Shah, H., Busbridge, D., Ali, A. M. E., Susskind, J., and Thilak, V. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models, 2025. URL https://arxiv.org/abs/2501.12370

    arXiv 2025

  3. [3]

    Unified scaling laws for routed language models, 2022

    Clark, A., de las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., Borgeaud, S., van den Driessche, G., Rutherford, E., Hennigan, T., Johnson, M., Millican, K., Cassirer, A., Jones, C., Buchatskaya, E., Budden, D., Sifre, L., Osindero, S., Vinyals, O., Rae, J., Elsen, E., Kavukcuoglu, K., and Simonyan, K. Unifi...

    work page 2022

  4. [4]

    X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y

    Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y. K., Huang, P., Luo, F., Ruan, C., Sui, Z., and Liang, W. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models, 2024

    work page 2024

  5. [5]

    DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., ...

    arXiv 2025

  6. [6]

    M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A

    Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., Fedus, L., Bosma, M., Zhou, Z., Wang, T., Wang, Y. E., Webster, K., Pellat, M., Robinson, K., Meier-Hellstern, K., Duke, T., Dixon, L., Zhang, K., Le, Q. V., Wu, Y., Chen, Z., and Cui, C. Glam: Efficient scaling of language models with mixt...

    work page 2022

  7. [7]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  8. [8]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

    Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

    work page 2022

Show all 41 references
  1. [9]

    Scaling laws for sparsely-connected foundation models, 2023

    Frantar, E., Riquelme, C., Houlsby, N., Alistarh, D., and Evci, U. Scaling laws for sparsely-connected foundation models, 2023

    2023

  2. [10]

    Megablocks: Efficient sparse training with mixture-of-experts, 2022

    Gale, T., Narayanan, D., Young, C., and Zaharia, M. Megablocks: Efficient sparse training with mixture-of-experts, 2022. URL https://arxiv.org/abs/2211.15841

    2022 arXiv

  3. [11]

    Scaling laws for neural machine translation, 2021

    Ghorbani, B., Firat, O., Freitag, M., Bapna, A., Krikun, M., Garcia, X., Chelba, C., and Cherry, C. Scaling laws for neural machine translation, 2021

    2021

  4. [12]

    Making deep learning go brrrr from first principles

    He, H. Making deep learning go brrrr from first principles. 2022. URL https://horace.io/brrr_intro.html

    2022

  5. [13]

    B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D

    Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., and McCandlish, S. Scaling laws for autoregressive generati...

    2020

  6. [14]

    Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically, 2017

    2017

  7. [15]

    and Schmidhuber, J

    Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9 0 (8): 0 1735--1780, 1997

    1997

  8. [16]

    A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vin...

    2022

  9. [17]

    B., Werra, L

    Hägele, A., Bakouch, E., Kosson, A., Allal, L. B., Werra, L. V., and Jaggi, M. Scaling laws and compute-optimal training beyond fixed training durations, 2024. URL https://arxiv.org/abs/2405.18392

    2024 arXiv

  10. [18]

    A., Jordan, M

    Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural Computation, 3 0 (1): 0 79--87, 1991. doi:10.1162/neco.1991.3.1.79

    1991 doi

  11. [19]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao...

    2024

  12. [20]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, 2020

    2020

  13. [21]

    F., Bordelon, B., Muennighoff, N., Paul, M., Pehlevan, C., Ré, C., and Raghunathan, A

    Kumar, T., Ankner, Z., Spector, B. F., Bordelon, B., Muennighoff, N., Paul, M., Pehlevan, C., Ré, C., and Raghunathan, A. Scaling laws for precision, 2024. URL https://arxiv.org/abs/2411.04330

    2024 arXiv

  14. [22]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.\ 1207--1216, Stanford, CA, 2000. Morgan Kaufmann

    2000

  15. [23]

    Gshard: Scaling giant models with conditional computation and automatic sharding, 2020

    Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding, 2020

    2020

  16. [24]

    Scaling laws for fine-grained mixture of experts

    Ludziejewski, J., Krajewski, J., Adamczewski, K., Pi\' o ro, M., Krutul, M., Antoniak, S., Ciebiera, K., Kr\' o l, K., Odrzyg\' o \' z d\' z , T., Sankowski, P., Cygan, M., and Jaszczur, S. Scaling laws for fine-grained mixture of experts. In Salakhutdinov, R., Kolter, Z., Hel...

    2024

  17. [25]

    McCandlish, S., Kaplan, J., Amodei, D., and Team, O. D. An empirical model of large-batch training, 2018. URL https://arxiv.org/abs/1812.06162

    2018 arXiv

  18. [26]

    A., Koh, P

    Muennighoff, N., Soldaini, L., Groeneveld, D., Lo, K., Morrison, J., Min, S., Shi, W., Walsh, P., Tafjord, O., Lambert, N., Gu, Y., Arora, S., Bhagia, A., Schwenk, D., Wadden, D., Wettig, A., Hui, B., Dettmers, T., Kiela, D., Farhadi, A., Smith, N. A., Koh, P. W., Singh, A., a...

    2024 arXiv

  19. [27]

    and Song, J

    Pearce, T. and Song, J. Reconciling kaplan and chinchilla scaling laws, 2024. URL https://arxiv.org/abs/2406.12907

    2024 arXiv

  20. [28]

    B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L

    Penedo, G., Kydlíček, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https://arxiv.org/abs/2406.17557

    2024 arXiv

  21. [29]

    Resolving discrepancies in compute-optimal scaling of language models, 2025

    Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Carmon, Y. Resolving discrepancies in compute-optimal scaling of language models, 2025. URL https://arxiv.org/abs/2406.19146

    2025 arXiv

  22. [30]

    Improving language understanding by generative pre-training

    Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training. 2018

    2018

  23. [31]

    Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., Welbl, J...

    2022

  24. [32]

    Beyond chinchilla-optimal: Accounting for inference in language model scaling laws, 2024

    Sardana, N., Portes, J., Doubov, S., and Frankle, J. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws, 2024. URL https://arxiv.org/abs/2401.00448

    2024 arXiv

  25. [33]

    Glu variants improve transformer, 2020

    Shazeer, N. Glu variants improve transformer, 2020. URL https://arxiv.org/abs/2002.05202

    2020 arXiv

  26. [34]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017

    2017

  27. [35]

    Mesh-tensorflow: Deep learning for supercomputers, 2018

    Shazeer, N., Cheng, Y., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P., Lee, H., Hong, M., Young, C., Sepassi, R., and Hechtman, B. Mesh-tensorflow: Deep learning for supercomputers, 2018

    2018

  28. [36]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/2104.09864

    2023 arXiv

  29. [37]

    Qwen2.5 technical report

    Team, Q. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024 a

    2024 arXiv

  30. [38]

    Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters", February 2024 b

    Team, Q. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters", February 2024 b . URL https://qwenlm.github.io/blog/qwen-moe/

    2024

  31. [39]

    P., and Zhang, H

    Yun, L., Zhuang, Y., Fu, Y., Xing, E. P., and Zhang, H. Toward inference-optimal mixture-of-expert large language models, 2024. URL https://arxiv.org/abs/2404.02852

    2024 arXiv

  32. [40]

    Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning

    Zadouri, T., \"U st \"u n, A., Ahmadian, A., Ermi s , B., Locatelli, A., and Hooker, S. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. arXiv preprint arXiv:2309.05444, 2023

    2023 arXiv

  33. [41]

    St-moe: Designing stable and transferable sparse expert models

    Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., and Fedus, W. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906, 2022

    2022 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.