{"id":"2124bf77-01d5-4737-a091-40a28be002b1","arxiv_id":"2608.10605","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":15,"one_line_summary":"Under a fixed model-FLOPs budget, the fitted loss for sparse MoE models decreases monotonically with sparsity, so the optimum sits at the boundary; an interior sparsity optimum appears only when hardware-deliverable FLOPs are used as the budget.","lead":"This paper combines a scaling law for sparse mixture-of-experts language models with a hardware performance model, selecting model size, sparsity, and parallel layout together for a fixed cluster and training window. It reports that optimizing sparsity against raw model FLOPs alone pushes to maximum sparsity, while accounting for throughput actually delivered by the cluster produces an interior, hardware-dependent optimum.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The no-interior-optimum claim is computed from one point on a flat identifiability ridge of Eq. (10); Appendix B.2 calls for joint ridge perturbations but none are run, so monotone L*(S) may be an artifact of one admissible fit.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the boundary-seeking and cluster-optimal conclusions are computed from Eq. (10), whose non-identified coefficients sit on a flat ridge. I agree with that diagnosis and see it as the most critical point because the no-interior-optimum statement is a derived property of the fitted law, not a directly measured property of the raw runs. The performance model is the other pillar, but it is validated in absolute MFU within 15% and in Top-5/Top-10 ranking overlap across three sweeps, and the staged ordering flip in Figure 12 is computed from measured MFU, so I do not see the systems model as the weaker link. A ridge-sensitivity check would settle whether the central claim is an artifact of a single admissible coefficient choice; until that check is run, a conditional verdict is appropriate. The paper deserves credit for releasing the ScalePlan code, reporting honest limitations, and running real staged pretraining runs, which is why I would not move the verdict to reject or unverified. Keeping the reader's CONDITIONAL verdict is the right adjustment.","tokens_in":40463,"tokens_out":6964,"duration_ms":86526,"concrete_test":"Re-run the 400-restart fit scan described in Appendix B.2; collect every coefficient vector whose Huber objective is within a small tolerance (e.g., 0.05 in log-loss units) of the Table 2 optimum. For each vector, solve Eq. (16) on a fine S grid from 0.50 to 0.981 at Cmodel in {3e19, 6e19, 1e20, 3e20, 6e20} and record the argmin S*. If any ridge-consistent fit has S* < 0.981, the boundary-seeking claim is not identified and Figure 4 should be reported as one admissible fit's behavior. If every sampled fit keeps S* at 0.981, the claim is robust to the admitted identifiability ridge.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central boundary-seeking result in Section 6.1 is not a direct empirical observation; it is the solution of Eq. (16) using the fitted law Eq. (10). Table 2 and Appendix B.2 state that only eta and beta are identified and that the remaining coefficients (a, alpha, c, lambda, j, delta, gamma) lie on a flat identifiability ridge. Appendix B.2 explicitly says downstream sensitivity studies should perturb those coefficients jointly along the ridge rather than one at a time, but no such study appears in the paper. Whether L*(S) decreases monotonically over S in [0.5, 0.981] is controlled by the balance between the capacity term a/N^alpha and the sparsity terms c/(1-S)^lambda and j/((1-S)^delta N^gamma G^eta) after eliminating D via C = 6(1-S)ND. Along the ridge, the same loss predictions can be consistent with larger lambda or delta paired with smaller gamma/alpha, which makes high-S configurations more expensive and can yield an interior S*. Because the headline claim is exactly the assertion that this does not happen, the ridge is load-bearing. Section 7.3 then compounds the risk by extrapolating about 150x beyond the largest fitted compute and to Ntot values roughly 5x beyond the fitted range. The paper's own Section 9 admits that uncertainty in the non-identified coefficients is not propagated through the discrete architecture search, so the no-interior-optimum and cluster-optimal results rest on an unexamined admissible region of the fit.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MOSAIC, a framework that couples a four-dimensional sparse-MoE scaling law (total parameters, sparsity, tokens, expert split factor) with an analytical, microbenchmark-calibrated performance model, and searches jointly over architecture and parallelism layout under a fixed hardware envelope. On roughly 150 from-scratch pretraining runs, the authors fit the scaling law, validate the performance model's MFU predictions on sweeps up to 18B active parameters, and report that an efficiency-agnostic model-FLOPs budget admits no interior sparsity optimum within the calibrated range, whereas a hardware-deliverable compute budget produces an interior cluster-optimal sparsity. Staged training runs up to 250B total parameters are used to check the predicted loss ordering on both model-FLOPs and hardware-compute axes.","tokens_in":40956,"tokens_out":2875,"duration_ms":57683,"significance":"If the central claims hold, the paper makes a genuinely useful contribution: it identifies a concrete failure mode of model-FLOPs-only scaling prescriptions for sparse MoEs, and it demonstrates a working architecture-systems co-design pipeline with a released performance estimator (ScalePlan), a reproducible scaling-law fit, and real validation runs. The empirical observation that loss ordering can flip between the model-FLOPs axis and the hardware-compute axis is valuable regardless of the exact fitted coefficients. However, the load-bearing no-interior-optimum result is computed from one point on a flat identifiability ridge of the fitted law, and the paper itself acknowledges that uncertainty in the non-identified coefficients is not propagated. The central claim is therefore defensible but not yet airtight; it requires a ridge-aware sensitivity analysis before it can be stated as a general property of sparse MoE scaling rather than a property of one admissible fit.","major_comments":[{"comment":"The no-interior-optimum claim is computed by solving Eq. (16) with the fitted law Eq. (10), but Table 2 and Appendix B.2 state that only eta and beta are identified and that the remaining coefficients lie on a wide, near-flat ridge. Appendix B.2 explicitly recommends that downstream sensitivity studies perturb these coefficients jointly along the ridge, yet no such study is reported. Since the monotonicity of L*(S) is controlled by the balance between the capacity term a/N^alpha and the sparsity terms c/(1-S)^lambda and j/((1-S)^delta N^gamma G^eta), an equally admissible ridge-consistent fit with a larger lambda or delta can produce an interior S*. Because the headline claim is exactly the assertion that no interior optimum exists, this identifiability issue is load-bearing and must be addressed with joint ridge perturbations or with an alternative functional form analysis.","section":"Section 6.1, Eq. (16), Table 2, Appendix B.2"},{"comment":"The MOSAIC optimum under the 32-node, 20-day envelope is reported at Cmodel values roughly 150x beyond the largest fitted compute budget, and Section 7.5's staged runs also operate 15-20x beyond the fitted regime. Section 9 concedes that some reported optima require substantial extrapolation. The boundary-seeking result and the cluster-optimal rankings both rely on the fitted law in this extrapolated region, where the form-imposed blow-up as S approaches 1 is not empirically measured (Appendix B.2). The authors should provide an intermediate-scale validation, or explicitly frame the results as extrapolations whose ranking confidence is not quantified.","section":"Section 7.3, Table 3, Section 7.3.1, Section 9"},{"comment":"The performance model's only fitted correction, chi_sys, is calibrated against measured iteration times but is not validated on held-out sweeps or ablated from the model. The reported MAPE under 15% validates the end-to-end MFU prediction, but it does not establish that chi_sys is physically identified rather than compensating for misspecifications elsewhere in the analytical model. Since chi_sys enters the feasibility constraint (C-FLOPs) and thus the MOSAIC ranking, the framework's sensitivity to this calibration should be characterized, for example by refitting on a subset of sweeps and testing on the remainder, or by reporting predictions with chi_sys fixed to 1.","section":"Section 5.5, Eq. (14)-(15), Section 7.2"},{"comment":"Figure 6 shows that the fitted law decreases monotonically with G at matched (Ntot, S, D), while the observed runs decrease up to G=4 and flatten or slightly reverse at G=8 within run noise. Since G is a searched dimension in MOSAIC and G=8 appears in the budget sweeps of Section 7.4, the law's extrapolation along G is not well supported by the observed data. The authors should either restrict the search to the G range where the law is empirically validated, or provide additional G=8 runs to resolve whether the predicted monotone advantage is real.","section":"Section 4.2, Figure 6"}],"minor_comments":[{"comment":"The manuscript contains numerous formatting typos, including 'Systems-A ware', 'F actor', 'SparsitySand', and 'T raining' in section headings. A careful proofreading pass is needed.","section":"Global"},{"comment":"Figure 1 states S_max = 0.985 in the left panel, while Table 6 reports the calibrated sparsity range as S in [0.499, 0.981]. The inconsistency should be resolved, since the boundary of the data support is central to the no-interior-optimum claim.","section":"Figure 1"},{"comment":"The reported optimal configuration is described as the exact optimum of the enumerated grid, but the token budget D is selected on a capped TPP grid of [40, 150]; the paper should state more explicitly that the optimum is exact only up to the resolution of that grid and the discrete search dimensions.","section":"Section 7.3, Table 4"},{"comment":"The distinction between the local BCa intervals in Table 2 and the true cross-basin spread along the ridge is well explained, but the paper should state in the main text, not only in the appendix, that the confidence intervals in Table 2 do not reflect the ridge uncertainty.","section":"Appendix B.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's own limitations section already concedes the two main concerns I raise: uncertainty in non-identified coefficients is not propagated, and the reported optima involve substantial extrapolation. The framework and empirical validation are strong enough to warrant a major revision rather than rejection, but the headline 'no interior optimum' claim should not appear as a settled empirical fact until the ridge sensitivity analysis is actually carried out. I would also encourage the editor to ask the authors to make the calibration and ridge-perturbation scripts available alongside ScalePlan, since reproducibility is a stated strength of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Strong empirical systems paper with a real new axis (G in the scaling law) and a calibrated MFU model in the loop. The headline \"no interior sparsity optimum\" is conditional on one non-identified fit, and the paper itself knows this; treat it as provisional.\n\nWhat is genuinely good: 150 from-scratch MoE pretraining runs spanning Ntot up to 79B, the joint law with expert split factor G, the boundary-seeking observation under model-FLOPs budgets, and the performance model validated under 15% MAPE across three sweeps. The released ScalePlan code and the staged runs that reproduce the predicted ordering flip on real hardware are concrete value. The limitations section (Sec. 9) is honest about extrapolation and unpropagated uncertainty.\n\nThe soft spot in proportion: the central no-interior-optimum claim comes from solving Eq. (16) with the fitted law Eq. (10), and Table 2 shows only eta and beta are identified. The remaining coefficients sit on a flat identifiability ridge, and Appendix B.2 says sensitivity should perturb them jointly along that ridge—but no such perturbation study appears. That leaves monotone L*(S) as one admissible solution, not a demonstrated property of the data. This is a genuine gap, not a manufactured one. The 150x compute extrapolation in Sec. 7.3 is a smaller concern because the paper leans on ordering rather than absolute loss, but it compounds the fragility. Minor: the performance model's only fitted term chi_sys is not shown to hold on held-out sweeps, and the staged-run ordering flip is between two configurations with margins inside the smoothed-curve width.\n\nWho it's for: people who actually pick MoE geometries on a fixed cluster. The framework and tooling are useful even if the boundary-seeking claim later turns out to be an artifact of the chosen functional form. A serious referee can get value here: ask for ridge-consistent perturbations of the scaling law (or a softened claim), and error bars on the staged comparison. This deserves peer review.","headline":"Strong systems-side contribution with real runs and code, but the headline no-interior-sparsity result depends on one non-identified fit; worth refereeing with a request to perturb the ridge.","tokens_in":41429,"tokens_out":3134,"would_cite":true,"duration_ms":35215,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that under a model-FLOPs budget, optimal MoE sparsity is boundary-seeking, and that a true interior sparsity optimum appears only when the budget is the hardware-deliverable FLOPs a cluster can actually produce, so…","keywords":["sparse mixture-of-experts","scaling laws","systems-aware co-design","model FLOPs utilization","parallelism layout","LLM pretraining","MOSAIC","boundary optimum"],"falsifier":"Extend the scaling-law dataset with runs at sparsity beyond the current support (for example $S=0.99$ and $S=0.995$), refit the joint law with alternative functional forms that let $G$ enter additively or with its own exponent per stratum, and solve the model-FLOPs-constrained minimization; if the fitted optimal loss $L^\\star(S)$ develops an interior minimum above $S=0.981$, the no-interior-optimum claim is overturned.","tokens_in":40267,"feed_emoji":"⚙️","tokens_out":7497,"duration_ms":70627,"temperature":0.7,"pith_summary":"This paper argues that the conventional way of choosing a sparse Mixture-of-Experts (MoE) language model—first pick the architecture and token budget by minimizing predicted loss under a model-FLOPs budget, then tune the implementation for hardware—is systematically misleading. Fitting a joint scaling law over total parameters, sparsity, tokens, and expert split factor to 150 pretraining runs, the authors find that within the calibrated sparsity range the predicted loss falls monotonically as sparsity rises, so a model-FLOPs budget has no interior optimum: it always says 'make the model as sparse as the data allow.' The paper then replaces the model-FLOPs budget with the deliverable FLOPs a fixed cluster actually produces, obtained by multiplying peak hardware FLOPs by the predicted Model FLOPs Utilization (MFU), and shows that under this budget an interior sparsity emerges that depends on the cluster size, GPU type, and training window. The proposed framework, MOSAIC, searches architecture, token budget, and parallel layout jointly, and staged real runs confirm that the loss ordering flips between the model-FLOPs axis and the hardware-compute axis. If correct, this reframes 'compute-optimal' scaling as a cluster-specific co-design problem rather than a FLOPs-only one.","feed_headline":"MoE sparsity is a cluster problem, not a FLOPs problem","feed_subtitle":"On a fixed cluster, the lowest-loss MoE geometry is set by attainable throughput, not by model FLOPs.","key_machinery":"The load-bearing object is the deliverable model FLOPs, $C_{\\mathrm{deliverable}} = C_{\\mathrm{peak}} \\times \\mathrm{MFU}(Z, P_\\ell, B) \\times \\eta_{\\mathrm{good}}(N_{\\mathrm{dev}})$, where $C_{\\mathrm{peak}}$ is the raw peak-FLOPs ceiling of the cluster over the training window and MFU is the fraction of that ceiling converted into model-math FLOPs under an execution plan. MOSAIC (Model Optimization via Systems-Aware Training Co-design) is a mixed-integer nonlinear program that minimizes the predicted loss $L(N_{\\mathrm{tot}}, S, D, G)$ from the joint scaling law subject to $C_{\\mathrm{model}} = 6 N_{\\mathrm{act}} D$ being no larger than $C_{\\mathrm{deliverable}}$, with the inner maximization over feasible parallel layouts handled by the released performance estimator ScalePlan. The scaling law is fitted on 150 runs with variables chosen to expose systems efficiency: total parameters $N_{\\mathrm{tot}}$, sparsity $S$ (the fraction of parameters inactive per token), tokens $D$, and expert split factor $G = d_{\\mathrm{ff}}/d_{\\mathrm{expert}}$ (the number of narrow experts per reference feed-forward network). Three structural properties make the bilevel program exactly solvable by enumeration: the loss is log-convex in its continuous variables, the non-convexity lives only in constraints, and the optimal token count $D^\\star$ saturates the constraint in closed form. The Gumbel proposal-correction routing recipe keeps the 150-run sweep stable enough that final-checkpoint losses are comparable fit targets.","core_discovery":"The central claim is stated directly in the abstract: within the calibrated sparsity range, an efficiency-agnostic model-FLOPs budget admits no interior optimal sparsity. The fitted loss decreases monotonically with sparser models and the compute optimum lies at the upper boundary of the data support; an optimal sparsity in MoE models instead emerges under the cluster's systems constraints, as captured by MOSAIC. The evidence is a four-dimensional scaling law $L(N_{\\mathrm{tot}},S,D,G)=\\frac{a}{N_{\\mathrm{tot}}^\\alpha}+\\frac{b}{D^\\beta}+\\frac{c}{(1-S)^\\lambda}+\\frac{j}{(1-S)^\\delta N_{\\mathrm{tot}}^\\gamma G^\\eta}+e$ fit to 150 from-scratch runs spanning 104M--2.7B active and up to 79B total parameters, coupled to an operator-level performance model that predicts MFU and memory and selects the best parallel layout. On a fixed 32-node, 20-day envelope, the lowest-loss configuration is not the one that emits the most model FLOPs: a $G=4$, $E=96$, $K=2$ model at $N_{\\mathrm{act}}=14.5\\mathrm{B}$ and $S=0.956$ reaches loss $1.3898$ on about 36% fewer model FLOPs than the highest-MFU candidate. The paper therefore concludes that ranking MoE configurations by realized model FLOPs misorders them, and that the right budget is the hardware-deliverable FLOPs.","pith_inferences":["A direct testable extension: re-run prior MoE scaling-law analyses (for example those that fix active parameters or report an interior optimal sparsity) under a fixed deliverable-FLOPs budget; the paper's boundary-versus-interior claim predicts their optima will shift toward sparser or denser geometries depending on the cluster.","The boundary-seeking result is conditional on the parametric form of the joint scaling law; because most of its coefficients sit on a flat identifiability ridge, the monotone descent in sparsity may not be a robust law, and sensitivity of the optimal sparsity to alternative functional forms is a necessary check the paper leaves open.","The MOSAIC formulation itself is not MoE-specific: the same coupling of a scaling law, a calibrated performance model, and a discrete architecture grid could be applied to dense transformers, hybrid architectures, or inference-time budgets, where the same model-FLOPs-versus-deliverable-FLOPs distinction should appear.","The fixed geometry ladder (one depth-to-width co-scaling family) constrains the search; allowing multiple ladders would reveal whether the plateau in optimal sparsity and the near-tie at $G=4$ are properties of the loss surface or artifacts of the single backbone family."],"forward_implications":["Under a pure model-FLOPs budget, any practitioner using this law will be pushed to the maximum supported sparsity; the correct sparsity for a given cluster and training window can only be found by pricing in the attainable MFU.","The same MoE geometry can rank first on one cluster and lower on another, because expert all-to-all crossing node boundaries, pipeline fill ratio, and memory pressure shift with device count and interconnect; MOSAIC's budget sweeps show the optimal sparsity stays on a plateau of roughly 0.915--0.963 while the expert split factor often ties near $G=4$.","Layout rules from the validated performance model---keep expert parallelism inside the NVLink domain, fix tensor parallelism to 1, and keep the pipeline fill ratio $r_{\\mathrm{fill}} = n_{\\mathrm{mb}}/P_{\\mathrm{PP}}$ at or above 4---give immediate throughput guidance for MoE training runs.","Comparisons between MoE architectures that count only model FLOPs can misrank them by a wide margin; dividing by measured or predicted MFU to plot loss against hardware compute restores the ordering, as the staged runs show.","Because the configuration search is exact over the realizable grid under the stated assumptions, MOSAIC's output is a concrete, launchable model plus parallel layout rather than a continuous prescription that must be discretized afterward."],"supporting_citations":[{"why":"Supplies the Chinchilla-style joint law in model size and tokens whose closed-form compute-optimal allocation the paper extends to sparsity and systems knobs.","marker":"Hoffmann et al. (2022)"},{"why":"Introduces sparsity as a scaling-law dimension and reports an extra sparsity optimum; the paper reproduces its three monotone trends on its own runs.","marker":"Abnar et al. (2025)"},{"why":"Introduces fine-grained MoE scaling with granularity as an explicit axis, motivating the expert split factor $G$ in the joint law.","marker":"Krajewski et al. (2024)"},{"why":"Finds that scaling expert count improves loss at fixed model FLOPs but inflates memory, providing the memory-aware pressure that motivates a hardware-deliverable budget.","marker":"Ludziejewski et al. (2025)"},{"why":"Argues that model FLOPs per token alone is an inadequate fairness metric for sparse MoE architecture choice, a premise the paper pushes further.","marker":"Wan et al. (2026)"},{"why":"Couples roofline analysis to architecture selection, the closest prior co-design approach whose overheads the paper identifies as incomplete.","marker":"Sun et al. (2026)"},{"why":"Provides the auxiliary-loss-free load balancing routing the training recipe builds on with its Gumbel proposal-correction modifications.","marker":"Wang et al. (2025b)"},{"why":"Defines the DeepSeekMoE layer with a shared expert and sequence-level auxiliary loss that the paper's MoE architecture follows.","marker":"DeepSeek-AI (2024)"}],"fun_headline_variants":["MoE sparsity is a hardware problem, not a FLOPs problem","Compute-optimal isn't cluster-optimal","No optimal MoE sparsity without cluster constraints","Best MoE geometry is set by your cluster, not by FLOPs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim that loss decreases monotonically with sparsity under a model-FLOPs budget is computed from a single fitted scaling-law form whose coefficients mostly lie on a flat identifiability ridge; if an equally admissible functional form produced a non-monotone loss in sparsity, the boundary-seeking conclusion would not hold.","fun_headline_variants_meta":{"raw":{"variants":["MoE sparsity is a hardware problem, not a FLOPs problem","Compute-optimal isn't cluster-optimal","No optimal MoE sparsity without cluster constraints","Best MoE geometry is set by your cluster, not by FLOPs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1556,"prompt_tokens":1176,"completion_tokens":380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":792,"completion_tokens_details":{"reasoning_tokens":310}},"tokens_in":792,"tokens_out":380,"duration_ms":3770,"temperature":1.0,"reasoning_tokens":310,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:02:44.835786+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Extend the scaling-law dataset with runs at sparsity beyond the current support (for example $S=0.99$ and $S=0.995$), refit the joint law with alternative functional forms that let $G$ enter additively or with its own exponent per stratum, and solve the model-FLOPs-constrained minimization; if the fitted optimal loss $L^\\star(S)$ develops an interior minimum above $S=0.981$, the no-interior-optimum claim is overturned.","supporting_citations":[],"review_version":1}