Pith. sign in

REVIEW 4 major objections 4 minor 59 references

Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that under a model-FLOPs budget, optimal MoE sparsity is boundary-seeking, and that a true interior sparsity optimum appears only when the budget is the hardware-deliverable FLOPs a cluster can actually produce, so…

desk verdict Strong systems-side contribution with real runs and code, but the headline no-interior-sparsity result depends on one non-identified fit; worth refereeing with a request to perturb the ridge. read the letter →

arxiv 2608.10605 v1 pith:JR3OG3MI submitted 2026-08-11 cs.LG cs.AI

Soumajyoti Sarkar , Yuxin Tang , Sheng Zha This is my paper · ORCID
classification cs.LGcs.AI
keywords sparsemixture-of-expertsscalinglawssystems-awareco-designmodelFLOPsutilizationparallelismlayoutLLMpretrainingMOSAICboundaryoptimum
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the conventional way of choosing a sparse Mixture-of-Experts (MoE) language model—first pick the architecture and token budget by minimizing predicted loss under a model-FLOPs budget, then tune the implementation for hardware—is systematically misleading. Fitting a joint scaling law over total parameters, sparsity, tokens, and expert split factor to 150 pretraining runs, the authors find that within the calibrated sparsity range the predicted loss falls monotonically as sparsity rises, so a model-FLOPs budget has no interior optimum: it always says 'make the model as sparse as the data allow.' The paper then replaces the model-FLOPs budget with the deliverable FLOPs a fixed cluster actually produces, obtained by multiplying peak hardware FLOPs by the predicted Model FLOPs Utilization (MFU), and shows that under this budget an interior sparsity emerges that depends on the cluster size, GPU type, and training window. The proposed framework, MOSAIC, searches architecture, token budget, and parallel layout jointly, and staged real runs confirm that the loss ordering flips between the model-FLOPs axis and the hardware-compute axis. If correct, this reframes 'compute-optimal' scaling as a cluster-specific co-design problem rather than a FLOPs-only one.

What carries the argument

The load-bearing object is the deliverable model FLOPs, $C_{\mathrm{deliverable}} = C_{\mathrm{peak}} \times \mathrm{MFU}(Z, P_\ell, B) \times \eta_{\mathrm{good}}(N_{\mathrm{dev}})$, where $C_{\mathrm{peak}}$ is the raw peak-FLOPs ceiling of the cluster over the training window and MFU is the fraction of that ceiling converted into model-math FLOPs under an execution plan. MOSAIC (Model Optimization via Systems-Aware Training Co-design) is a mixed-integer nonlinear program that minimizes the predicted loss $L(N_{\mathrm{tot}}, S, D, G)$ from the joint scaling law subject to $C_{\mathrm{model}} = 6 N_{\mathrm{act}} D$ being no larger than $C_{\mathrm{deliverable}}$, with the inner maximization over feasible parallel layouts handled by the released performance estimator ScalePlan. The scaling law is fitted on 150 runs with variables chosen to expose systems efficiency: total parameters $N_{\mathrm{tot}}$, sparsity $S$ (the fraction of parameters inactive per token), tokens $D$, and expert split factor $G = d_{\mathrm{ff}}/d_{\mathrm{expert}}$ (the number of narrow experts per reference feed-forward network). Three structural properties make the bilevel program exactly solvable by enumeration: the loss is log-convex in its continuous variables, the non-convexity lives only in constraints, and the optimal token count $D^\star$ saturates the constraint in closed form. The Gumbel proposal-correction routing recipe keeps the 150-run sweep stable enough that final-checkpoint losses are comparable fit targets.

What would settle it

Extend the scaling-law dataset with runs at sparsity beyond the current support (for example $S=0.99$ and $S=0.995$), refit the joint law with alternative functional forms that let $G$ enter additively or with its own exponent per stratum, and solve the model-FLOPs-constrained minimization; if the fitted optimal loss $L^\star(S)$ develops an interior minimum above $S=0.981$, the no-interior-optimum claim is overturned.

Watch

Extended reading notes

Core claim

The central claim is stated directly in the abstract: within the calibrated sparsity range, an efficiency-agnostic model-FLOPs budget admits no interior optimal sparsity. The fitted loss decreases monotonically with sparser models and the compute optimum lies at the upper boundary of the data support; an optimal sparsity in MoE models instead emerges under the cluster's systems constraints, as captured by MOSAIC. The evidence is a four-dimensional scaling law $L(N_{\mathrm{tot}},S,D,G)=\frac{a}{N_{\mathrm{tot}}^\alpha}+\frac{b}{D^\beta}+\frac{c}{(1-S)^\lambda}+\frac{j}{(1-S)^\delta N_{\mathrm{tot}}^\gamma G^\eta}+e$ fit to 150 from-scratch runs spanning 104M--2.7B active and up to 79B total parameters, coupled to an operator-level performance model that predicts MFU and memory and selects the best parallel layout. On a fixed 32-node, 20-day envelope, the lowest-loss configuration is not the one that emits the most model FLOPs: a $G=4$, $E=96$, $K=2$ model at $N_{\mathrm{act}}=14.5\mathrm{B}$ and $S=0.956$ reaches loss $1.3898$ on about 36% fewer model FLOPs than the highest-MFU candidate. The paper therefore concludes that ranking MoE configurations by realized model FLOPs misorders them, and that the right budget is the hardware-deliverable FLOPs.

Load-bearing premise

The central claim that loss decreases monotonically with sparsity under a model-FLOPs budget is computed from a single fitted scaling-law form whose coefficients mostly lie on a flat identifiability ridge; if an equally admissible functional form produced a non-monotone loss in sparsity, the boundary-seeking conclusion would not hold.

Editorial extensions

If this is right

  • Under a pure model-FLOPs budget, any practitioner using this law will be pushed to the maximum supported sparsity; the correct sparsity for a given cluster and training window can only be found by pricing in the attainable MFU.
  • The same MoE geometry can rank first on one cluster and lower on another, because expert all-to-all crossing node boundaries, pipeline fill ratio, and memory pressure shift with device count and interconnect; MOSAIC's budget sweeps show the optimal sparsity stays on a plateau of roughly 0.915--0.963 while the expert split factor often ties near $G=4$.
  • Layout rules from the validated performance model---keep expert parallelism inside the NVLink domain, fix tensor parallelism to 1, and keep the pipeline fill ratio $r_{\mathrm{fill}} = n_{\mathrm{mb}}/P_{\mathrm{PP}}$ at or above 4---give immediate throughput guidance for MoE training runs.
  • Comparisons between MoE architectures that count only model FLOPs can misrank them by a wide margin; dividing by measured or predicted MFU to plot loss against hardware compute restores the ordering, as the staged runs show.
  • Because the configuration search is exact over the realizable grid under the stated assumptions, MOSAIC's output is a concrete, launchable model plus parallel layout rather than a continuous prescription that must be discretized afterward.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension: re-run prior MoE scaling-law analyses (for example those that fix active parameters or report an interior optimal sparsity) under a fixed deliverable-FLOPs budget; the paper's boundary-versus-interior claim predicts their optima will shift toward sparser or denser geometries depending on the cluster.
  • The boundary-seeking result is conditional on the parametric form of the joint scaling law; because most of its coefficients sit on a flat identifiability ridge, the monotone descent in sparsity may not be a robust law, and sensitivity of the optimal sparsity to alternative functional forms is a necessary check the paper leaves open.
  • The MOSAIC formulation itself is not MoE-specific: the same coupling of a scaling law, a calibrated performance model, and a discrete architecture grid could be applied to dense transformers, hybrid architectures, or inference-time budgets, where the same model-FLOPs-versus-deliverable-FLOPs distinction should appear.
  • The fixed geometry ladder (one depth-to-width co-scaling family) constrains the search; allowing multiple ladders would reveal whether the plateau in optimal sparsity and the near-tie at $G=4$ are properties of the loss surface or artifacts of the single backbone family.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MOSAIC, a framework that couples a four-dimensional sparse-MoE scaling law (total parameters, sparsity, tokens, expert split factor) with an analytical, microbenchmark-calibrated performance model, and searches jointly over architecture and parallelism layout under a fixed hardware envelope. On roughly 150 from-scratch pretraining runs, the authors fit the scaling law, validate the performance model's MFU predictions on sweeps up to 18B active parameters, and report that an efficiency-agnostic model-FLOPs budget admits no interior sparsity optimum within the calibrated range, whereas a hardware-deliverable compute budget produces an interior cluster-optimal sparsity. Staged training runs up to 250B total parameters are used to check the predicted loss ordering on both model-FLOPs and hardware-compute axes.

Significance. If the central claims hold, the paper makes a genuinely useful contribution: it identifies a concrete failure mode of model-FLOPs-only scaling prescriptions for sparse MoEs, and it demonstrates a working architecture-systems co-design pipeline with a released performance estimator (ScalePlan), a reproducible scaling-law fit, and real validation runs. The empirical observation that loss ordering can flip between the model-FLOPs axis and the hardware-compute axis is valuable regardless of the exact fitted coefficients. However, the load-bearing no-interior-optimum result is computed from one point on a flat identifiability ridge of the fitted law, and the paper itself acknowledges that uncertainty in the non-identified coefficients is not propagated. The central claim is therefore defensible but not yet airtight; it requires a ridge-aware sensitivity analysis before it can be stated as a general property of sparse MoE scaling rather than a property of one admissible fit.

major comments (4)
  1. [Section 6.1, Eq. (16), Table 2, Appendix B.2] The no-interior-optimum claim is computed by solving Eq. (16) with the fitted law Eq. (10), but Table 2 and Appendix B.2 state that only eta and beta are identified and that the remaining coefficients lie on a wide, near-flat ridge. Appendix B.2 explicitly recommends that downstream sensitivity studies perturb these coefficients jointly along the ridge, yet no such study is reported. Since the monotonicity of L*(S) is controlled by the balance between the capacity term a/N^alpha and the sparsity terms c/(1-S)^lambda and j/((1-S)^delta N^gamma G^eta), an equally admissible ridge-consistent fit with a larger lambda or delta can produce an interior S*. Because the headline claim is exactly the assertion that no interior optimum exists, this identifiability issue is load-bearing and must be addressed with joint ridge perturbations or with an alternative functional form analysis.
  2. [Section 7.3, Table 3, Section 7.3.1, Section 9] The MOSAIC optimum under the 32-node, 20-day envelope is reported at Cmodel values roughly 150x beyond the largest fitted compute budget, and Section 7.5's staged runs also operate 15-20x beyond the fitted regime. Section 9 concedes that some reported optima require substantial extrapolation. The boundary-seeking result and the cluster-optimal rankings both rely on the fitted law in this extrapolated region, where the form-imposed blow-up as S approaches 1 is not empirically measured (Appendix B.2). The authors should provide an intermediate-scale validation, or explicitly frame the results as extrapolations whose ranking confidence is not quantified.
  3. [Section 5.5, Eq. (14)-(15), Section 7.2] The performance model's only fitted correction, chi_sys, is calibrated against measured iteration times but is not validated on held-out sweeps or ablated from the model. The reported MAPE under 15% validates the end-to-end MFU prediction, but it does not establish that chi_sys is physically identified rather than compensating for misspecifications elsewhere in the analytical model. Since chi_sys enters the feasibility constraint (C-FLOPs) and thus the MOSAIC ranking, the framework's sensitivity to this calibration should be characterized, for example by refitting on a subset of sweeps and testing on the remainder, or by reporting predictions with chi_sys fixed to 1.
  4. [Section 4.2, Figure 6] Figure 6 shows that the fitted law decreases monotonically with G at matched (Ntot, S, D), while the observed runs decrease up to G=4 and flatten or slightly reverse at G=8 within run noise. Since G is a searched dimension in MOSAIC and G=8 appears in the budget sweeps of Section 7.4, the law's extrapolation along G is not well supported by the observed data. The authors should either restrict the search to the G range where the law is empirically validated, or provide additional G=8 runs to resolve whether the predicted monotone advantage is real.
minor comments (4)
  1. [Global] The manuscript contains numerous formatting typos, including 'Systems-A ware', 'F actor', 'SparsitySand', and 'T raining' in section headings. A careful proofreading pass is needed.
  2. [Figure 1] Figure 1 states S_max = 0.985 in the left panel, while Table 6 reports the calibrated sparsity range as S in [0.499, 0.981]. The inconsistency should be resolved, since the boundary of the data support is central to the no-interior-optimum claim.
  3. [Section 7.3, Table 4] The reported optimal configuration is described as the exact optimum of the enumerated grid, but the token budget D is selected on a capped TPP grid of [40, 150]; the paper should state more explicitly that the optimum is exact only up to the resolution of that grid and the discrete search dimensions.
  4. [Appendix B.2] The distinction between the local BCa intervals in Table 2 and the true cross-basin spread along the ridge is well explained, but the paper should state in the main text, not only in the appendix, that the confidence intervals in Table 2 do not reflect the ridge uncertainty.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pipeline is fit-then-optimize with independent hardware validation; identifiability caveats are correctness risks, not circular reasoning.

full rationale

The paper's central claims are produced by fitting the joint scaling law of Eq. (10) to 150 real pretraining runs and then solving the constrained optimizations of Eqs. (16) and (C-FLOPs). The boundary-seeking sparsity result is an argmin of that fitted law, not a parameter that was fit as an input, so it is not a fitted quantity being renamed as a prediction. Likewise, MOSAIC's cluster-optimal sparsity is the output of a mixed-integer program whose objective is the same fitted loss and whose feasibility constraint uses a performance model calibrated against measured kernel and collective microbenchmarks and validated with under 15% MFU error. The staged validation runs in Section 7.5 compare measured loss and measured MFU, so the observed ordering flip between the model-FLOPs and hardware-compute axes is an external check rather than a consequence of the model's own definitions. The only self-citation (Sarkar et al., 2024) is a background pointer for MoE architecture adoption and is not load-bearing. The paper explicitly acknowledges that most scaling-law coefficients lie on a flat identifiability ridge and that coefficient uncertainty is not propagated through the search (Section 9, Appendix B.2); this is a robustness and extrapolation limitation affecting the confidence in any single optimum, but it does not make the derivation circular. No step in the derivation reduces by definition or by self-citation to its own inputs.

Assumptions & free parameters 15 free parameters · 6 assumptions · 0 invented entities

The central claims rest on a fitted four-parameter-plus-exponents scaling law, a calibrated MFU model with one fitted correction, and a hand-chosen geometry ladder and routing recipe. No new physical entities are introduced. The free-parameter count is high relative to the 150 training runs, and several scaling law coefficients are explicitly on a flat ridge.

free parameters (15)
  • scaling law prefactor a = 8.462
    Fitted in Eq. (10); marked as ridge in Table 2.
  • scaling law exponent alpha = 0.1048
    Capacity exponent in Eq. (10); sits on the identifiability ridge.
  • scaling law prefactor b = 88.32
    Data term prefactor in Eq. (10); marked as ridge.
  • scaling law exponent beta = 0.2070
    Data exponent in Eq. (10); reported as identified.
  • sparsity floor prefactor c = 0.1828
    Stand-alone sparsity floor in Eq. (10); marked as ridge.
  • sparsity floor exponent lambda = 0.1249
    Sparsity floor exponent in Eq. (10); marked as ridge.
  • interaction prefactor j = 0.7123
    Joint sparsity-capacity-G term in Eq. (10); marked as ridge.
  • interaction sparsity exponent delta = 0.5567
    Sparsity exponent in the joint term; marked as ridge.
  • interaction capacity exponent gamma = 0.1702
    Capacity exponent in the joint term; marked as ridge.
  • expert split exponent eta = 0.9513
    G exponent in Eq. (10); reported as identified with BCa interval [+0.91, +0.95].
  • irreducible loss e = 0.3129
    Constant term in Eq. (10); marked as ridge.
  • learning-rate law prefactor and exponent = 3.30, -0.199
    Fitted on a pilot sweep in Eq. (17) and used to assign peak learning rates for all runs.
  • chi_sys correction coefficients c_A2A, c_PP, c_ovl = not reported
    The performance model's only fitted term, introduced in Eq. (14)-(15); values and fitting procedure are not disclosed.
  • geometry ladder constants = m=4, d_ff=2.5*d_model, d_head=128
    Hand-chosen co-scaling in Section 7.1; all searched architectures and the optimum are conditional on this ladder.
  • routing and recipe constants = tau=0.01, rho=12, mu_b=1e-3, z-loss coeff 1e-4, seq aux coeff 1e-3
    Hand-chosen hyperparameters in Section 3.3 and Appendix A.3, kept fixed across all runs.
assumptions (6)
  • domain assumption The four-term posynomial scaling law in Eq. (10) is the correct loss surface for sparse MoE models over the searched range
    The boundary-sparsity result and all MOSAIC loss ranks are computed from this fitted form; Section 4.2 presents it as a modeling choice, and Appendix B.2 shows most coefficients sit on an identifiability ridge.
  • domain assumption The fitted scaling law extrapolates far beyond its training compute range
    Section 7.3 states the returned optimum realizes model FLOPs about 150x beyond the largest fit run, supported only by the law's assumed form.
  • domain assumption The performance model with additive compute, dispatch, pipeline, and vocabulary terms plus a single correction chi_sys captures cluster throughput
    Sections 5.4 and 5.5; MOSAIC's feasibility constraint and token budget rely on the absolute MFU and memory predictions.
  • ad hoc to paper The geometry ladder co-scaling family fixes L_layers=4q, n_head=4q, d=128*n_head, and d_ff=2.5*d
    Section 7.1 and Table 9; all searched architectures and the returned optimum are conditional on this ladder.
  • ad hoc to paper The LFLB routing modifications with Gumbel proposal-correction, row normalization, and RMS bias controller keep training stable
    Section 3.3 and Algorithm 1; the recipe is introduced for this sweep and is not a general routing law.
  • domain assumption A single pass over the mC4 corpus with final-checkpoint smoothed loss is an adequate proxy for the loss surface
    Sections 3.1 and Appendix A.1; all scaling law fits assume this training setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts." pith.science (2026). https://pith.science/paper/JR3OG3MI

@misc{pith2026260810605,
  author       = {Pith},
  title        = {Pith review of: Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JR3OG3MI}},
  note         = {Machine review of arXiv:2608.10605}
}
abstract

In large-scale pretraining, the algorithm, architecture, and systems decisions are conventionally made in disconnected stages. A scaling law stage selects an architecture and training recipe, optimizing loss under compute constraints, and a separate systems stage then optimizes the implementation for hardware efficiency. In this work, we develop MOSAIC, which formulates model architecture and systems co-design as an optimization problem. MOSAIC couples a predictive scaling law with a calibrated performance model that estimates Model FLOPs Utilization (MFU), communication cost, memory footprint, and the best parallel layout. We instantiate the framework for sparse Mixture-of-Experts (MoE) language models, where expert count, routing sparsity, and other MoE layer dimensions affect both the loss and systems efficiency. We fit a scaling law on sparse MoE models trained on text data, whose scaling dimensions include the sparsity factor, which is the fraction of model parameters inactive per token in a forward pass. The scaling law sweeps in our work span active parameters from $104$ million to $2.7$ billion and total model sizes reaching $79$ billion parameters. We show that, within the calibrated sparsity range, an efficiency-agnostic model-FLOPs budget admits no interior optimal sparsity. The fitted loss decreases monotonically with sparser models and the compute optimum lies at the upper boundary of the data support. An optimal sparsity in MoE models instead emerges under the cluster's systems constraints, as captured by MOSAIC. Our results argue for a shift towards unified architecture and systems co-design for frontier language model training.

Figures

Figures reproduced from arXiv: 2608.10605 by the authors.

Figure 1
Figure 1. The systems-aware scaling motivation for MOSAIC. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The MOSAIC workflow. It takes the device configuration and the cluster size and training window as inputs, queries the geometry scaling ladder for candidate geometries, the performance model for predicted peak MFU, and the scaling law for predicted loss, and returns the model configuration together with its parallel layout. We start with describing the abstract formulation of MOSAIC in Section 2. To apply MOSAIC wit… view at source ↗
Figure 3
Figure 3. Predicted-vs-observed training loss for the L(Ntot, S, D) fit in Equation (9), fit per expert split factor. The panels for every stratum we fit, G∈ {1, 2, 4, 8}, are in [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Model FLOPs optimal N⋆ tot, N⋆ act, and L ⋆ as a function of sparsity S, at five budgets C. The curves are computed from our own G= 4 fit of Equation (9) on our runs. Larger MoE sparsity S increases optimal total parameters, decreases optimal active parameters, and low…
Figure 5
Figure 5. Figure 5: Loss against total MoE parameters Ntot. In the left, one curve per expert split factor G at fixed S = 0.95, D = 6 B. In the right, one curve per sparsity band at fixed G= 4, D = 6 B. The curves fan out rather than staying parallel (left), which is why G enters the law …
Figure 6
Figure 6. Figure 6: Training loss against expert split factor G = dff/dexpert at matched (Ntot, S, D) operating points. The predicted loss decreases monotonically with G. The observed runs decrease up to G= 4 and flatten or slightly reverse at G= 8 within run noise. The gap between the co…
Figure 7
Figure 7. Figure 7: Joint-law parity plot over all runs, R2 = 0.99877. The crosses are the held-out validation runs, the largest 10% by Nact within each G stratum, with a held-out RMSE of 0.00779 in loss units. Marker shape and color pick out a sample of the (G, K) pairs the sweep covers.…
Figure 8
Figure 8. Figure 8: Opposing views of model scaling and systems cost. The top row varies MoE sparsity S holding G= 4. The bottom row varies the expert split factor G holding S = 0.958. The left column gives the optimal loss under a fixed model-FLOPs budget. Each marker there is labeled wi…
Figure 9
Figure 9. Figure 9: Predicted versus Real MFU per parallelism configuration, each panel normalized to its own maximum and sorted by real MFU. 7.2 Performance Model Validation Our validation spans three sweeps on these p6 nodes, over the q= 4, q= 8, and q= 12 rungs of [PITH_FULL_IMAGE:fig…
Figure 10
Figure 10. Figure 10: Loss versus model FLOPs Cmodel over all feasible (Z, D) configurations under the 32-node, 20-day envelope, color-coded by G. Every candidate draws on the same raw hardware ceiling of 9.95 × 1023 hardware FLOPs. The gold star is the optimal configuration with MOSAIC, a…
Figure 11
Figure 11. Figure 11: Optimal predicted loss vs. the budget under the two sweeps. Markers are colored by the optimal expert split factor G⋆ and annotated with S ⋆ and N⋆ tot. Dashed lines denote best per G configuration. The solid line is the overall optimum. The optimal sparsity S ⋆ does …
Figure 12
Figure 12. Figure 12: Smoothed loss trajectories of the staged runs against model FLOPs Cmodel (left) and against the peak-equivalent hardware compute consumed Cmodel/MFU (right), with the warm-up discarded. ID 2 leads over much of the lower model￾FLOPs range, while the trajectories conver…
Figure 13
Figure 13. Figure 13: Setting the peak learning rate. Left: loss-vs-learning-rate curves across MoE configurations, each with a quadratic-fit optimum. Right: those optima collapse onto a single power law in the model FLOPs Cmodel, with one law predicting the optimal learning rate across th…
Figure 14
Figure 14. Figure 14: Example of an MoE layer in a transformer block, at three expert split factors G∈ {1, 4, 8}. In every panel, the routed experts sit beside a shared expert. dff is the hidden width of a single reference dense feed-forward expert (G = 1), and the total routed model width…
Figure 15
Figure 15. Figure 15: Predicted-vs-observed loss for the L(Ntot, S, D) fit with Equation (9), refit in￾dependently on each expert split factor stratum G∈ {1, 2, 4, 8}. The G= 4 and G= 8 panels are the ones reproduced in [PITH_FULL_IMAGE:figures/full_fig_p045_15.png]
Figure 16
Figure 16. Figure 16: Training dynamics of the four largest MoE models in our scaling run sweep (annotated with Ntot, Nact, E, K, G). Each panel stacks, over training steps, the loss, gradient norm, starved-expert count, and per-expert bias range and mean. Loss descends smoothly, starved e…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 22 canonical work pages

  1. [1]

    arXiv preprint arXiv:2001.08361 , year=

    Scaling laws for neural language models , author=. arXiv preprint arXiv:2001.08361 , year=

    arXiv 2001

  2. [2]

    arXiv preprint arXiv:2203.15556 , year=

    Training compute-optimal large language models , author=. arXiv preprint arXiv:2203.15556 , year=

    arXiv

  3. [3]

    OpenAI blog , volume=

    Language models are unsupervised multitask learners , author=. OpenAI blog , volume=

  4. [4]

    arXiv preprint arXiv:2405.18392 , year=

    Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations , author=. arXiv preprint arXiv:2405.18392 , year=

    arXiv

  5. [5]

    arXiv preprint arXiv:2402.07871 , year=

    Scaling laws for fine-grained mixture of experts , author=. arXiv preprint arXiv:2402.07871 , year=

    arXiv

  6. [6]

    The Journal of Machine Learning Research , volume=

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity , author=. The Journal of Machine Learning Research , volume=. 2022 , publisher=

    2022

  7. [7]

    arXiv preprint arXiv:2602.17004 , year=

    Arcee trinity large technical report , author=. arXiv preprint arXiv:2602.17004 , year=

    arXiv

  8. [8]

    arXiv preprint arXiv:2401.06066 , year=

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models , author=. arXiv preprint arXiv:2401.06066 , year=

    arXiv

Show all 59 references
  1. [9]

    arXiv preprint arXiv:2507.20534 , year=

    Kimi k2: Open agentic intelligence , author=. arXiv preprint arXiv:2507.20534 , year=

    arXiv

  2. [10]

    arXiv preprint arXiv:1701.06538 , year=

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer , author=. arXiv preprint arXiv:1701.06538 , year=

    arXiv

  3. [11]

    arXiv preprint arXiv:2303.06182 , year=

    Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference , author=. arXiv preprint arXiv:2303.06182 , year=

    arXiv

  4. [12]

    arXiv preprint arXiv:2409.01483 , year=

    Revisiting smoe language models by evaluating inefficiencies with task specific expert pruning , author=. arXiv preprint arXiv:2409.01483 , year=

    arXiv

  5. [13]

    arXiv preprint arXiv:1712.00409 , year=

    Deep learning scaling is predictable, empirically , author=. arXiv preprint arXiv:1712.00409 , year=

    arXiv

  6. [14]

    21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) , pages=

    \ MegaScale \ : Scaling large language model training to more than 10,000 \ GPUs \ , author=. 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) , pages=

  7. [15]

    International Conference on Learning Representations , year=

    Towards greater leverage: Scaling laws for efficient mixture-of-experts language models , author=. International Conference on Learning Representations , year=

  8. [16]

    arXiv preprint arXiv:2306.03745 , year=

    Soft merging of experts with adaptive routing , author=. arXiv preprint arXiv:2306.03745 , year=

    arXiv

  9. [17]

    Proceedings of the National Academy of Sciences , volume=

    Explaining neural scaling laws , author=. Proceedings of the National Academy of Sciences , volume=. 2024 , publisher=

    2024

  10. [18]

    arXiv preprint arXiv:2202.08906 , year=

    St-moe: Designing stable and transferable sparse expert models , author=. arXiv preprint arXiv:2202.08906 , year=

    arXiv

  11. [19]

    arXiv preprint arXiv:2010.11934 , year=

    mT5: A massively multilingual pre-trained text-to-text transformer , author=. arXiv preprint arXiv:2010.11934 , year=

    2010 arXiv

  12. [20]

    arXiv preprint arXiv:1711.05101 , year=

    Decoupled weight decay regularization , author=. arXiv preprint arXiv:1711.05101 , year=

    arXiv

  13. [21]

    International Conference on Machine Learning , pages=

    Unified scaling laws for routed language models , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  14. [22]

    International Conference on Machine Learning , pages=

    Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  15. [23]

    Advances in Neural Information Processing Systems , volume=

    Kinetics: Rethinking Test-Time Scaling Law , author=. Advances in Neural Information Processing Systems , volume=

  16. [24]

    arXiv preprint arXiv:1909.08053 , year=

    Megatron-lm: Training multi-billion parameter language models using model parallelism , author=. arXiv preprint arXiv:1909.08053 , year=

    1909 arXiv

  17. [25]

    arXiv preprint arXiv:2410.06511 , year=

    TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training , author=. arXiv preprint arXiv:2410.06511 , year=

    arXiv

  18. [26]

    Proceedings of the 37th International Conference on Supercomputing , pages=

    A hybrid tensor-expert-data parallelism approach to optimize mixture-of-experts training , author=. Proceedings of the 37th International Conference on Supercomputing , pages=

  19. [27]

    Advances in Neural Information Processing Systems , volume=

    Mixture-of-experts with expert choice routing , author=. Advances in Neural Information Processing Systems , volume=

  20. [28]

    arXiv preprint arXiv:2512.16248 , year=

    Sigma-MoE-Tiny Technical Report , author=. arXiv preprint arXiv:2512.16248 , year=

  21. [29]

    Proceedings of Machine Learning and Systems , volume=

    MegaBlocks: Efficient Sparse Training with Mixture-of-Experts , author=. Proceedings of Machine Learning and Systems , volume=

  22. [30]

    Sparse Backpropagation for

    Liu, Liyuan and Gao, Jianfeng and Chen, Weizhu , journal=. Sparse Backpropagation for

  23. [31]

    International Conference on Learning Representations (ICLR) , year=

    Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts , author=. International Conference on Learning Representations (ICLR) , year=

  24. [32]

    , journal=

    Yellott, John I. , journal=. The relationship between. 1977 , publisher=

    1977

  25. [33]

    Stochastic Beams and Where to Find Them: The

    Kool, Wouter and van Hoof, Herke and Welling, Max , booktitle=. Stochastic Beams and Where to Find Them: The. 2019 , publisher=

    2019

  26. [34]

    Parameters vs

    Abnar, Samira and Shah, Harshay and Busbridge, Dan and Ali, Alaaeldin Mohamed Elnouby and Susskind, Josh and Thilak, Vimal , journal=. Parameters vs

  27. [35]

    Optimization and Engineering , volume=

    A tutorial on geometric programming , author=. Optimization and Engineering , volume=. 2007 , publisher=

    2007

  28. [36]

    arXiv preprint arXiv:2411.13055 , year=

    Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training , author=. arXiv preprint arXiv:2411.13055 , year=

    arXiv

  29. [37]

    Proceedings of the 53rd International Conference on Parallel Processing (ICPP '24) , year=

    The Case for Co-Designing Model Architectures with Hardware , author=. Proceedings of the 53rd International Conference on Parallel Processing (ICPP '24) , year=

  30. [38]

    arXiv preprint arXiv:2501.06589 , year=

    Ladder-residual: parallelism-aware architecture for accelerating large model inference with communication overlapping , author=. arXiv preprint arXiv:2501.06589 , year=

    arXiv

  31. [39]

    arXiv preprint arXiv:2408.00724 , year=

    Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models , author=. arXiv preprint arXiv:2408.00724 , year=

    arXiv

  32. [40]

    Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC) , year=

    Calculon: a Methodology and Tool for High-Level Co-Design of Systems and Large Language Models , author=. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC) , year=

  33. [41]

    2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO) , pages=

    vtrain: A simulation framework for evaluating cost-effective and compute-optimal large language model training , author=. 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO) , pages=. 2024 , organization=

    2024

  34. [42]

    arXiv preprint arXiv:2501.18107 , year=

    Scaling Inference-Efficient Language Models , author=. arXiv preprint arXiv:2501.18107 , year=

    arXiv

  35. [43]

    Scaling Laws Meet Model Architecture: Toward Inference-Efficient

    Bian, Song and Yu, Tao and Venkataraman, Shivaram and Park, Youngsuk , journal=. Scaling Laws Meet Model Architecture: Toward Inference-Efficient

  36. [44]

    Zheng, Size and Zheng, Xuegui and Chang, Li-wen and Zhai, Jidong , journal=

  37. [45]

    What Language Model to Train if You Have One Million

    Le Scao, Teven and Wang, Thomas and Hesslow, Daniel and Saulnier, Lucile and Bekman, Stas and Bari, M Saiful and Biderman, Stella and Elsahar, Hady and Muennighoff, Niklas and Phang, Jason and Press, Ofir and Raffel, Colin and Sanh, Victor and Shen, Sheng and Sutawika, Lintang...

  38. [46]

    arXiv preprint arXiv:2605.01640 , year=

    Prescriptive scaling laws for data constrained training , author=. arXiv preprint arXiv:2605.01640 , year=

    arXiv

  39. [47]

    arXiv preprint arXiv:2510.04800 , year=

    Hybrid architectures for language models: Systematic analysis and design insights , author=. arXiv preprint arXiv:2510.04800 , year=

    arXiv

  40. [48]

    Wang, Irene and Ardalani, Newsha and Elhoushi, Mostafa and Jiang, Daniel and Hsia, Samuel and Sumbul, Ekin and Mahajan, Divya and Wu, Carole-Jean and Acun, Bilge , journal=

  41. [49]

    arXiv preprint arXiv:2603.21862 , year=

    Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization , author=. arXiv preprint arXiv:2603.21862 , year=

  42. [50]

    Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device

    Sun, Luoyang and Jiang, Jiwen and Ding, Yifeng and Li, Fengfa and Song, Yan and Zhang, Haifeng and Ying, Jian and Ren, Lei and Zhan, Kun and Chen, Wei and Xie, Yan and Deng, Cheng , journal=. Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device

  43. [51]

    Ludziejewski, Jan and Pi. Joint. Proceedings of the 42nd International Conference on Machine Learning (ICML) , year=

  44. [52]

    arXiv preprint arXiv:2502.06857 , year=

    Gemstones: A Model Suite for Multi-Faceted Scaling Laws , author=. arXiv preprint arXiv:2502.06857 , year=

  45. [53]

    arXiv preprint arXiv:2505.06708 , year=

    Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free , author=. arXiv preprint arXiv:2505.06708 , year=

    arXiv

  46. [54]

    arXiv preprint arXiv:2412.19437 , year=

    arXiv

  47. [55]

    arXiv preprint arXiv:2601.20205 , year=

    Hyperparameter Transfer with Mixture-of-Expert Layers , author=. arXiv preprint arXiv:2601.20205 , year=

    arXiv

  48. [56]

    Advances in Neural Information Processing Systems , year=

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness , author=. Advances in Neural Information Processing Systems , year=

  49. [57]

    arXiv preprint arXiv:2603.05451 , year=

    Flashattention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling , author=. arXiv preprint arXiv:2603.05451 , year=

  50. [58]

    Journal of machine learning research , volume=

    Palm: Scaling language modeling with pathways , author=. Journal of machine learning research , volume=

  51. [59]

    Proceedings of the 52nd Annual International Symposium on Computer Architecture , year=

    Scaling Llama 3 Training with Efficient Parallelism Strategies , author=. Proceedings of the 52nd Annual International Symposium on Computer Architecture , year=

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.