{"id":"fc984bc9-e76b-42d5-b7c7-5b34ffff3b79","arxiv_id":"2608.03457","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Scaling laws for MoE diffusion language models are measured and used to train LLaDA MoE v2 30B-A3B, which approaches Qwen3 on several benchmarks with 65% of the training tokens.","lead":"MoE diffusion language models get their own scaling rules: batch size should grow faster and learning rate decay faster with compute than in autoregressive models, and optimal compute tilts slightly toward more data. The authors train a 30B-A3B model on 23.5T tokens using these rules and reach near-Qwen3 performance on several benchmarks with 35% fewer tokens.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Architecture sweeps violate the fixed-FLOP condition: varying expert count changes the router FLOPs counted in Eq. 8, so the A/G scaling trends may reflect unequal compute rather than architecture.","rationale":"The reader's weakest assumption is extrapolation of the fitted scaling laws from ~1e20 to ~5e23 FLOPs and the lack of an intermediate-scale check. That is a valid concern, but I find a more specific, internal problem that is load-bearing for the architecture half of the central claim. The paper's Appendix A.5 claims that the activated model-side budget M*(C) is preserved across architecture candidates 'ignoring the negligible router contribution,' yet Eq. 8 includes exactly that router cost, and the sweeps vary the number of experts n_e by large factors. This means the A-sweep and G-sweep are not IsoFLOP-controlled comparisons; candidates with more experts consume more compute per token. Since the loss differences that select the optimum are small, a 1.5–7% compute advantage can be material, and the paper provides no control for it. The shared-expert-ratio sweep fixes n_e, so its S=33.3% conclusion is not affected by this particular issue. The large-model benchmark results and the compute-versus-performance comparison with LLaDA MoE give genuine evidence of practical value, and I would not reject the paper. However, the architecture-scaling conclusions should be re-verified with exact FLOP matching before they are asserted as scaling laws. Because the reader already returned CONDITIONAL, my concern does not change the verdict; it sharpens one of the conditions that should be met. I therefore mark agreement as 'disagree' because the reader did not identify this specific fixed-compute violation, even though our overall verdicts coincide.","tokens_in":24768,"tokens_out":6996,"duration_ms":67318,"concrete_test":"Re-run the activation-ratio and expert-granularity sweeps at the two largest reference budgets (2e19 and 6e19 FLOPs) while holding the full per-token FLOPs M from Eq. 8 exactly constant across candidates—for example, adjust d_model or the training-token budget D per candidate so that every A/G configuration has the same total compute including the 2*d_model*n_e router term. Then compare the lowest-loss A and G values and the resulting recommended (A, G, S) for LLaDA MoE v2. If the sparsest-A optimum moves to a higher A, or G=8–16 is no longer the robust plateau, the architecture-scaling conclusion fails and the 9.09% activation-ratio choice is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's architecture-scaling claims—sparser activation favored at larger scales and G=8–16 robust—rest on sweeps that are stated to hold the activated model-side budget M*(C) fixed (Section 3.3, Appendix A.5). They do not. The compute accounting in Appendix A.1, Eq. 8, includes the router term 2*d_model*n_e in M, and both the activation-ratio sweep (vary n_e with n_a, n_s, d_expert fixed) and the expert-granularity sweep (n_e scales with G) change n_e by up to 128x and 10x across candidates. Appendix A.5 explicitly says the router contribution is 'negligible' and is ignored when preserving M*(C), but using the paper's own configurations, the router term differs across candidates by roughly 130k FLOPs/token at the 6e17 budget and by hundreds of thousands of FLOPs/token at the 6e19 budgets, corresponding to roughly 1.5–7% of total per-token compute depending on sweep and scale. Because every candidate is trained for the same 3D*(C) tokens, low-A and high-G candidates actually receive more total compute, so their lower loss could reflect extra FLOPs rather than a genuinely better architecture. This directly threatens the activation-ratio trend that motivates LLaDA MoE v2's 9.09% activation ratio and the G=8–16 robustness conclusion. The large-model results are real evidence of practical value, but they are not a controlled test of the architecture sweeps, since the deployed architecture was selected using those same sweeps.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a scaling-law study for mixture-of-experts diffusion language models (MoE dLLMs). It reports fitted power laws for optimal nominal batch size and learning rate as functions of compute, an IsoFLOP allocation law between activated model-side FLOPs and training tokens, and architecture sweeps over activation ratio, expert granularity, and shared-expert ratio at fixed activated budgets. It then trains LLaDA MoE v2, a 30B-A3B diffusion MoE, from scratch on 23.5T tokens using design choices derived from these laws, and reports benchmarks showing it approaches Qwen3 30B-A3B on several tasks with roughly 65% as many pretraining tokens, and that after SFT it outperforms SDAR Chat on seven of eight reasoning/coding benchmarks. The central empirical claims are that MoE dLLMs have dLLM-specific scaling laws quantitatively different from AR models, with a characteristic data-side allocation tilt and stable S=33.3% shared-expert optimum.","tokens_in":25152,"tokens_out":9318,"duration_ms":74331,"significance":"If the fitted laws are correct, they are practically useful design rules for scaling MoE dLLMs, and the paper would be among the first to characterize MoE dLLM scaling rather than inheriting AR heuristics. The paper's strengths include the breadth of controlled sweeps, the detailed configuration tables in the appendices that support reproducibility, the compute-controlled comparison with LLaDA MoE 7B-A1B in Figure 5, and the large-scale 30B-A3B training run with a clear SFT evaluation. The main risks are statistical: the headline exponent differences are reported without confidence intervals, and the architecture sweeps do not maintain exactly equal total compute because of the router FLOPs term, so the activation-ratio and granularity trends may be partly confounded. The large-scale results are a valuable case study but do not by themselves independently validate the exponents given the extrapolation gap.","major_comments":[{"comment":"The architecture sweeps in §3.3 and Appendix A.5 do not actually hold the activated model-side budget fixed, because the compute accounting in Eq. (8) includes the router term 2*d_model*n_e in M. In the activation-ratio sweep (Table 11), n_e ranges from 2 to 256, and in the expert-granularity sweep (Table 12), n_e ranges from 64 to 640. Using the paper's own configurations, the router term varies across candidates by roughly 3M FLOPs/token at the 6×10^17 budget and by roughly 18M FLOPs/token at the 6×10^19 budget, corresponding to up to about 2–5% of total per-token compute depending on the sweep and budget. Since every candidate is trained for 3D*(C) tokens, total compute C=MD is not equalized: low-A and high-G candidates systematically receive more FLOPs, so the lower losses observed for these configurations in Figure 4(a,b) could reflect extra compute rather than a genuinely superior architecture. The statement in Appendix A.5 that the router contribution is 'negligible' is not consistent with these numbers. This directly affects the design choices A=9.09% and G=8 adopted for LLaDA MoE v2 in §4. The authors should either re-run the sweeps with total compute exactly matched (for example, by adjusting token counts to absorb router cost) or provide a quantitative analysis demonstrating that the trends survive the correction.","section":"§3.3, Appendix A.5, Eq. (8)"},{"comment":"The quantitative claim that MoE dLLM scaling differs from AR scaling rests on comparisons of fitted exponents, e.g., the batch-size exponent 0.3481 versus DeepSeek's 0.3271 in Eq. (2) and Figure 1, and the allocation exponents 0.475/0.525 in Eq. (3) and Figure 3. These exponents are obtained by log–log linear regression over a small number of compute budgets (Tables 5 and 9), but no standard errors, confidence intervals, or goodness-of-fit statistics are reported anywhere in the manuscript. The difference of 0.021 between the batch exponents is small relative to the scatter visible in Figure 1, so without uncertainty quantification the paper does not establish a statistically significant difference from the AR reference laws. Please report confidence intervals (or bootstrap intervals) for all fitted exponents and for the implied M*/D* frontiers, and adjust the strength of the 'quantitative differences' claims accordingly.","section":"§3.1, Eq. (2), Figure 1; §3.2, Eq. (3), Figure 3"},{"comment":"The scaling laws are fitted at compute budgets up to 1×10^20 FLOPs for hyperparameters (Appendix A.3) and 6×10^19 FLOPs for architecture (Appendix A.5), while LLaDA MoE v2 is trained at roughly 5×10^23 FLOPs. The single large-model run in Section 4 is not a controlled test of any individual exponent: its hyperparameters, allocation, and architecture are all read off the same fitted laws, so agreement does not validate the laws independently, and no intermediate-scale check above roughly 6×10^20 FLOPs is reported. The large-model benchmark results are practically valuable, but they should be presented as a case study consistent with the laws rather than as evidence that the fitted exponents extrapolate. I would ask the authors to either add an intermediate-scale controlled check for at least the allocation and activation-ratio dimensions or explicitly soften the abstract/conclusion claims from 'establish practical scaling laws' to a conditional statement.","section":"§4, Appendix A.3, Appendix A.5"}],"minor_comments":[{"comment":"The caption says 'Colors indicate the activated model-side budget M*(C)', but no colorbar or legend is visible in the figure; please add one.","section":"Figure 4"},{"comment":"The statement that the activation-ratio drift in the shared-expert-ratio sweep is 'negligible' is not quantified; given that the S=33.3% optimum is a headline claim, please report the actual A values for each candidate.","section":"Appendix A.5, shared-expert-ratio sweep"},{"comment":"The text says the joint search is extended 'from 3×10^20 to 6×10^20 FLOPs' for Figure 2, but Table 5 lists the largest 3.6B-model budget as 3×10^20; please clarify the actual compute budget of the runs in Figure 2.","section":"§3.1 and Table 5"},{"comment":"The headline 'seven of eight' comparison against SDAR Chat mixes numbers reported in prior papers with numbers from the unified evaluation; please state explicitly which entries are newly measured under the protocol in Appendix B.4.","section":"Table 4 and Appendix B.4"},{"comment":"Please state at the equations that C is measured in non-embedding training FLOPs and that B* is a nominal token batch size, since these units are central to interpreting the coefficients.","section":"Eqs. (2) and (3)"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to attract attention given the scale of the training run and the practical framing. The router-accounting issue is fixable with reanalysis rather than new training, and I would not reject on that basis. I would also encourage the editor to ask for the code and sweep logs, since the quantitative conclusions hinge on regression fits that are not currently reproducible from the paper alone."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. This is the first systematic scaling-law study for MoE diffusion language models, and it ships a 30B-A3B validation model that gets reasonably close to Qwen3 with 35% fewer pretraining tokens. That part is real and useful. But the architecture-scaling claims, especially the activation-ratio trend that motivates the 9.09% choice, are confounded: the sweeps do not actually hold activated FLOPs fixed, because the router cost 2*d_model*n_e is allowed to vary. The paper acknowledges this in Appendix A.5 but calls it negligible; using their own Eq. 8, the router term differs by up to ~7% of per-token compute across candidates. Since every candidate trains on the same 3D*(C) tokens, the low-A and high-G candidates get strictly more FLOPs. So the lower loss could be a compute effect, not an architecture effect. That's a real soft spot in the central claim.\n\nWhat's genuinely good: the hyperparameter and allocation sweeps are extensive and transparently documented; the IsoFLOP methodology is standard; the fitted laws are concrete and usable. The large model is a serious scale-up and the benchmark table is honest about where it lags Qwen3. The compute-controlled comparison against LLaDA MoE 7B is a nice practical validation.\n\nAlso soft: the headline exponents have no uncertainty estimates, and the extrapolation from ~1e20 to ~5e23 FLOPs is a leap validated by one model run. That's common in this literature, but it means the quantitative differences from AR laws (batch exponent 0.3481 vs 0.3271) should be read as suggestive, not established. The paper also doesn't ship code or data, though the appendix has enough detail to reconstruct the sweeps.\n\nWho this is for: anyone working on dLLM scaling or MoE architecture. It deserves serious referee time; I'd send it out, but the architecture section needs to either redo the sweeps with the router term included in the fixed budget, or show the loss differences survive a FLOP-adjusted comparison. As it stands, the allocation and hyperparameter laws are the more solid contributions; the architecture recommendations are plausible but not yet proven.","headline":"First systematic scaling study for MoE dLLMs with a real 30B validation run, but the architecture sweeps are confounded by unaccounted router FLOPs, so treat the activation-ratio trend as plausible rather than proven.","tokens_in":25748,"tokens_out":2255,"would_cite":true,"duration_ms":20145,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"Mixture-of-experts diffusion language models obey their own compute-scaling laws.","keywords":["diffusion language models","mixture of experts","scaling laws","IsoFLOP analysis","compute allocation","shared experts","architecture scaling","pretraining"],"falsifier":"Train checkpoints at an intermediate budget around $10^{21}$ to $10^{22}$ FLOPs—between the fitting range and the flagship—and vary batch size, learning rate, token/model allocation, and shared-expert ratio around the law's predictions; if any off-law configuration beats the predicted optimum by more than the 0.25% loss margin used in fitting, the extrapolated laws are falsified.","tokens_in":24548,"feed_emoji":"📈","tokens_out":8645,"duration_ms":67480,"temperature":0.7,"pith_summary":"Mixture-of-experts (MoE) diffusion language models—models that denoise masked tokens with sparsely activated experts—have their own compute-scaling behavior, and it differs systematically from autoregressive (AR) scaling. The paper fits power laws for the optimal nominal batch size and learning rate, for the split of compute between activated model capacity and training tokens, and for the MoE architecture itself. The fitted laws are $B^* = 0.374\\,C^{0.3481}$, $\\eta^* = 64.8\\,C^{-0.2447}$, $M^* = 0.5152\\,C^{0.475}$, and $D^* = 1.9411\\,C^{0.525}$, with a stable shared-expert optimum at $S = 33.3\\%$. A 30B-A3B model trained under these rules, LLaDA MoE v2, approaches a leading autoregressive 30B-A3B model on several benchmarks while using about 65% of its pretraining tokens. A sympathetic reader would care because, if the laws hold, MoE dLLM design stops being guesswork and becomes a set of compute-optimal recipes.","feed_headline":"MoE diffusion LLMs get their own scaling laws","feed_subtitle":"Fitted power laws set batch size, learning rate, token budget, and shared-expert ratio.","key_machinery":"The load-bearing object is a compute accounting identity plus a three-parameter architecture decomposition. Compute is counted as $C = M D$, where $M$ is activated non-embedding FLOPs per token and $D$ is the number of nominal training tokens; this replaces the dense-model $6ND$ approximation and is what lets sparse models be compared fairly. On top of that the paper defines three architectural axes: activation ratio $A=(n_a+n_s)/(n_e+n_s)$ (fraction of expert capacity activated per token), expert granularity $G=2d_{\\mathrm{model}}/d_{\\mathrm{expert}}$ (how routed capacity is split into experts), and shared-expert ratio $S=n_s/(n_a+n_s)$ (fraction of activated capacity going to a shared pathway). The argument is carried by controlled small-scale sweeps—joint batch-size/learning-rate search, IsoFLOP allocation sweeps, and one-at-a-time architecture sweeps at fixed activated budget—each fitted with power laws, whose exponents then become the design rule for the 30B run.","core_discovery":"The paper's central claim is that mixture-of-experts diffusion language models follow their own, quantitatively distinct scaling laws, and that obeying them is what lets a large dLLM compete with strong autoregressive models at reduced token cost. Fitted on controlled sweeps from $10^{17}$ to $3\\times10^{20}$ FLOPs, the laws say the optimal nominal batch size grows faster with compute than in AR models ($B^* = 0.374\\,C^{0.3481}$), the optimal learning rate decays faster ($\\eta^* = 64.8\\,C^{-0.2447}$), and compute-optimal allocation is nearly balanced with a slight data-side tilt ($M^* = 0.5152\\,C^{0.475}$ for activated FLOPs per token, $D^* = 1.9411\\,C^{0.525}$ for tokens). For architecture, the optimal share of activated capacity held by shared experts stays at $S=33.3\\%$ across scales, in contrast to AR MoE designs whose shared-expert fraction shrinks. The paper validates these rules by training LLaDA MoE v2, a 30B-total/3B-activated model, on 23.5T tokens: it approaches Qwen3 30B-A3B on several knowledge, reasoning, and coding benchmarks with about 65% as many pretraining tokens, and after standard SFT it outperforms SDAR Chat 30B-A3B on seven of eight reasoning and coding tasks.","pith_inferences":["If the fitted exponents hold beyond the fitting range, the data-side tilt (0.525 vs 0.475) predicts that at every larger budget the cheapest loss reduction comes from adding training tokens, so data collection and filtering should be scaled ahead of activated model size.","The shared-expert optimum at $S=33.3\\%$ is an architectural statement, not a consequence of the power-law fits; a direct test would be to train a second large model with the same $A$ and $G$ but $S=25\\%$ or $S=0\\%$ and compare loss at equal FLOPs.","The batch-size law's steeper exponent may reflect the masked objective's roughly 50% effective prediction rate; if one redefined $C$ in terms of expected masked tokens, the dLLM and AR laws might collapse onto a common curve, a unification the paper does not claim.","Because the flagship run is a single point, the cleanest validation is an intermediate-scale law check around $10^{21}$–$10^{22}$ FLOPs before committing to a $5\\times10^{23}$ FLOPs run."],"forward_implications":["At any compute budget $C$, the laws give a concrete recipe: set nominal batch size to $0.374C^{0.3481}$, learning rate to $64.8C^{-0.2447}$, and allocate compute so that activated FLOPs per token and tokens follow $M^* = 0.5152C^{0.475}$ and $D^* = 1.9411C^{0.525}$.","AR scaling laws are useful priors but not portable: MoE dLLMs need larger nominal batches and smaller learning rates than DeepSeek-style AR laws predict at the same compute.","The optimal allocation tilts toward data: the token budget grows faster than activated model-side computation, so larger MoE dLLM runs should spend marginal compute on more tokens rather than more activated capacity.","Architecturally, larger scales favor sparser activation (smaller $A$), moderate expert granularity $G=8$–$16$ works across scales, and the active capacity fraction given to shared experts stays at one third.","The 30B-A3B flagship trained under these rules approaches an autoregressive 30B-A3B baseline with roughly 65% of the baseline's pretraining tokens, and after SFT alone it beats a diffusion-autoregressive hybrid on seven of eight reasoning and coding benchmarks."],"supporting_citations":[{"why":"Supplies the scaling-law methodology and the compute-as-6ND accounting that the paper adapts to MoE.","marker":"Kaplan et al., 2020"},{"why":"Introduces IsoFLOP analysis for model-data allocation, the method used to fit $M^*(C)$ and $D^*(C)$.","marker":"Hoffmann et al., 2022"},{"why":"Provides the DeepSeek LLM AR batch-size and learning-rate laws used as reference curves in Figure 1.","marker":"Bi et al., 2024"},{"why":"Reports AR MoE scaling laws and a decreasing optimal shared-expert ratio, the contrast for the stable $S=33.3\\%$ result.","marker":"Tian et al., 2026"},{"why":"Defines the fine-grained expert parameterization of $G$ and $S$ that the paper adopts.","marker":"Krajewski et al., 2024"},{"why":"Analyzes parameters versus FLOPs for MoE sparsity, informing the activation-ratio treatment.","marker":"Abnar et al., 2025"},{"why":"Provides joint MoE scaling laws and the $C=MD$ compute accounting used for sparse models.","marker":"Ludziejewski et al., 2025"},{"why":"Prior MoE dLLM without scaling-law guidance; serves as the compute-matched baseline in Figure 5.","marker":"Zhu et al., 2025"},{"why":"Qwen3 30B-A3B is the main autoregressive baseline for the token-efficiency comparison.","marker":"Yang et al., 2025"},{"why":"SMDM dense dLLM scaling results give the dense diffusion reference frontier in Table 2.","marker":"Nie et al., 2025"}],"fun_headline_variants":["Diffusion MoE scaling laws differ from autoregressive","LLaDA MoE v2: scaling laws that beat AR with fewer tokens","MoE dLLMs: new scaling laws, faster batch growth","Shared experts stay put: scaling MoE diffusion models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that power laws measured on models up to 3.6B parameters and about $10^{20}$ FLOPs continue to hold at the 30B scale and roughly $5\\times10^{23}$ FLOPs of the flagship run.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion MoE scaling laws differ from autoregressive","LLaDA MoE v2: scaling laws that beat AR with fewer tokens","MoE dLLMs: new scaling laws, faster batch growth","Shared experts stay put: scaling MoE diffusion models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000328,"raw_usage":{"total_tokens":1919,"prompt_tokens":1117,"completion_tokens":802,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":733,"completion_tokens_details":{"reasoning_tokens":730}},"tokens_in":733,"tokens_out":802,"duration_ms":6701,"temperature":1.0,"reasoning_tokens":730,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:49:29.569428+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train checkpoints at an intermediate budget around $10^{21}$ to $10^{22}$ FLOPs—between the fitting range and the flagship—and vary batch size, learning rate, token/model allocation, and shared-expert ratio around the law's predictions; if any off-law configuration beats the predicted optimum by more than the 0.25% loss margin used in fitting, the extrapolated laws are falsified.","supporting_citations":[],"review_version":2}