Pith. sign in

REVIEW 3 major objections 5 minor 90 references

LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mixture-of-experts diffusion language models obey their own compute-scaling laws.

desk verdict First systematic scaling study for MoE dLLMs with a real 30B validation run, but the architecture sweeps are confounded by unaccounted router FLOPs, so treat the activation-ratio trend as plausible rather than proven. read the letter →

arxiv 2608.03457 v1 pith:27GB7ZRJ submitted 2026-08-04 cs.AI

Fengqi Zhu , Shaoxuan Xu , Jingyang Ou , Zebin You , Yipeng Xing , Huabin Liu , Xiaolu Zhang , Jun Zhou
show 6 more authors
Zhenzhong Lan Yankai Lin Wayne Xin Zhao Jianguo Li Chongxuan Li Ji-Rong Wen
This is my paper · ORCID
classification cs.AI MSC 68T0768T50
keywords diffusionlanguagemodelsmixtureofexpertsscalinglawsIsoFLOPanalysiscomputeallocationsharedarchitecturepretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mixture-of-experts (MoE) diffusion language models—models that denoise masked tokens with sparsely activated experts—have their own compute-scaling behavior, and it differs systematically from autoregressive (AR) scaling. The paper fits power laws for the optimal nominal batch size and learning rate, for the split of compute between activated model capacity and training tokens, and for the MoE architecture itself. The fitted laws are $B^* = 0.374\,C^{0.3481}$, $\eta^* = 64.8\,C^{-0.2447}$, $M^* = 0.5152\,C^{0.475}$, and $D^* = 1.9411\,C^{0.525}$, with a stable shared-expert optimum at $S = 33.3\%$. A 30B-A3B model trained under these rules, LLaDA MoE v2, approaches a leading autoregressive 30B-A3B model on several benchmarks while using about 65% of its pretraining tokens. A sympathetic reader would care because, if the laws hold, MoE dLLM design stops being guesswork and becomes a set of compute-optimal recipes.

What carries the argument

The load-bearing object is a compute accounting identity plus a three-parameter architecture decomposition. Compute is counted as $C = M D$, where $M$ is activated non-embedding FLOPs per token and $D$ is the number of nominal training tokens; this replaces the dense-model $6ND$ approximation and is what lets sparse models be compared fairly. On top of that the paper defines three architectural axes: activation ratio $A=(n_a+n_s)/(n_e+n_s)$ (fraction of expert capacity activated per token), expert granularity $G=2d_{\mathrm{model}}/d_{\mathrm{expert}}$ (how routed capacity is split into experts), and shared-expert ratio $S=n_s/(n_a+n_s)$ (fraction of activated capacity going to a shared pathway). The argument is carried by controlled small-scale sweeps—joint batch-size/learning-rate search, IsoFLOP allocation sweeps, and one-at-a-time architecture sweeps at fixed activated budget—each fitted with power laws, whose exponents then become the design rule for the 30B run.

What would settle it

Train checkpoints at an intermediate budget around $10^{21}$ to $10^{22}$ FLOPs—between the fitting range and the flagship—and vary batch size, learning rate, token/model allocation, and shared-expert ratio around the law's predictions; if any off-law configuration beats the predicted optimum by more than the 0.25% loss margin used in fitting, the extrapolated laws are falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that mixture-of-experts diffusion language models follow their own, quantitatively distinct scaling laws, and that obeying them is what lets a large dLLM compete with strong autoregressive models at reduced token cost. Fitted on controlled sweeps from $10^{17}$ to $3\times10^{20}$ FLOPs, the laws say the optimal nominal batch size grows faster with compute than in AR models ($B^* = 0.374\,C^{0.3481}$), the optimal learning rate decays faster ($\eta^* = 64.8\,C^{-0.2447}$), and compute-optimal allocation is nearly balanced with a slight data-side tilt ($M^* = 0.5152\,C^{0.475}$ for activated FLOPs per token, $D^* = 1.9411\,C^{0.525}$ for tokens). For architecture, the optimal share of activated capacity held by shared experts stays at $S=33.3\%$ across scales, in contrast to AR MoE designs whose shared-expert fraction shrinks. The paper validates these rules by training LLaDA MoE v2, a 30B-total/3B-activated model, on 23.5T tokens: it approaches Qwen3 30B-A3B on several knowledge, reasoning, and coding benchmarks with about 65% as many pretraining tokens, and after standard SFT it outperforms SDAR Chat 30B-A3B on seven of eight reasoning and coding tasks.

Load-bearing premise

Everything rests on the assumption that power laws measured on models up to 3.6B parameters and about $10^{20}$ FLOPs continue to hold at the 30B scale and roughly $5\times10^{23}$ FLOPs of the flagship run.

Editorial extensions

If this is right

  • At any compute budget $C$, the laws give a concrete recipe: set nominal batch size to $0.374C^{0.3481}$, learning rate to $64.8C^{-0.2447}$, and allocate compute so that activated FLOPs per token and tokens follow $M^* = 0.5152C^{0.475}$ and $D^* = 1.9411C^{0.525}$.
  • AR scaling laws are useful priors but not portable: MoE dLLMs need larger nominal batches and smaller learning rates than DeepSeek-style AR laws predict at the same compute.
  • The optimal allocation tilts toward data: the token budget grows faster than activated model-side computation, so larger MoE dLLM runs should spend marginal compute on more tokens rather than more activated capacity.
  • Architecturally, larger scales favor sparser activation (smaller $A$), moderate expert granularity $G=8$–$16$ works across scales, and the active capacity fraction given to shared experts stays at one third.
  • The 30B-A3B flagship trained under these rules approaches an autoregressive 30B-A3B baseline with roughly 65% of the baseline's pretraining tokens, and after SFT alone it beats a diffusion-autoregressive hybrid on seven of eight reasoning and coding benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the fitted exponents hold beyond the fitting range, the data-side tilt (0.525 vs 0.475) predicts that at every larger budget the cheapest loss reduction comes from adding training tokens, so data collection and filtering should be scaled ahead of activated model size.
  • The shared-expert optimum at $S=33.3\%$ is an architectural statement, not a consequence of the power-law fits; a direct test would be to train a second large model with the same $A$ and $G$ but $S=25\%$ or $S=0\%$ and compare loss at equal FLOPs.
  • The batch-size law's steeper exponent may reflect the masked objective's roughly 50% effective prediction rate; if one redefined $C$ in terms of expected masked tokens, the dLLM and AR laws might collapse onto a common curve, a unification the paper does not claim.
  • Because the flagship run is a single point, the cleanest validation is an intermediate-scale law check around $10^{21}$–$10^{22}$ FLOPs before committing to a $5\times10^{23}$ FLOPs run.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents a scaling-law study for mixture-of-experts diffusion language models (MoE dLLMs). It reports fitted power laws for optimal nominal batch size and learning rate as functions of compute, an IsoFLOP allocation law between activated model-side FLOPs and training tokens, and architecture sweeps over activation ratio, expert granularity, and shared-expert ratio at fixed activated budgets. It then trains LLaDA MoE v2, a 30B-A3B diffusion MoE, from scratch on 23.5T tokens using design choices derived from these laws, and reports benchmarks showing it approaches Qwen3 30B-A3B on several tasks with roughly 65% as many pretraining tokens, and that after SFT it outperforms SDAR Chat on seven of eight reasoning/coding benchmarks. The central empirical claims are that MoE dLLMs have dLLM-specific scaling laws quantitatively different from AR models, with a characteristic data-side allocation tilt and stable S=33.3% shared-expert optimum.

Significance. If the fitted laws are correct, they are practically useful design rules for scaling MoE dLLMs, and the paper would be among the first to characterize MoE dLLM scaling rather than inheriting AR heuristics. The paper's strengths include the breadth of controlled sweeps, the detailed configuration tables in the appendices that support reproducibility, the compute-controlled comparison with LLaDA MoE 7B-A1B in Figure 5, and the large-scale 30B-A3B training run with a clear SFT evaluation. The main risks are statistical: the headline exponent differences are reported without confidence intervals, and the architecture sweeps do not maintain exactly equal total compute because of the router FLOPs term, so the activation-ratio and granularity trends may be partly confounded. The large-scale results are a valuable case study but do not by themselves independently validate the exponents given the extrapolation gap.

major comments (3)
  1. [§3.3, Appendix A.5, Eq. (8)] The architecture sweeps in §3.3 and Appendix A.5 do not actually hold the activated model-side budget fixed, because the compute accounting in Eq. (8) includes the router term 2*d_model*n_e in M. In the activation-ratio sweep (Table 11), n_e ranges from 2 to 256, and in the expert-granularity sweep (Table 12), n_e ranges from 64 to 640. Using the paper's own configurations, the router term varies across candidates by roughly 3M FLOPs/token at the 6×10^17 budget and by roughly 18M FLOPs/token at the 6×10^19 budget, corresponding to up to about 2–5% of total per-token compute depending on the sweep and budget. Since every candidate is trained for 3D*(C) tokens, total compute C=MD is not equalized: low-A and high-G candidates systematically receive more FLOPs, so the lower losses observed for these configurations in Figure 4(a,b) could reflect extra compute rather than a genuinely superior architecture. The statement in Appendix A.5 that the router contribution is 'negligible' is not consistent with these numbers. This directly affects the design choices A=9.09% and G=8 adopted for LLaDA MoE v2 in §4. The authors should either re-run the sweeps with total compute exactly matched (for example, by adjusting token counts to absorb router cost) or provide a quantitative analysis demonstrating that the trends survive the correction.
  2. [§3.1, Eq. (2), Figure 1; §3.2, Eq. (3), Figure 3] The quantitative claim that MoE dLLM scaling differs from AR scaling rests on comparisons of fitted exponents, e.g., the batch-size exponent 0.3481 versus DeepSeek's 0.3271 in Eq. (2) and Figure 1, and the allocation exponents 0.475/0.525 in Eq. (3) and Figure 3. These exponents are obtained by log–log linear regression over a small number of compute budgets (Tables 5 and 9), but no standard errors, confidence intervals, or goodness-of-fit statistics are reported anywhere in the manuscript. The difference of 0.021 between the batch exponents is small relative to the scatter visible in Figure 1, so without uncertainty quantification the paper does not establish a statistically significant difference from the AR reference laws. Please report confidence intervals (or bootstrap intervals) for all fitted exponents and for the implied M*/D* frontiers, and adjust the strength of the 'quantitative differences' claims accordingly.
  3. [§4, Appendix A.3, Appendix A.5] The scaling laws are fitted at compute budgets up to 1×10^20 FLOPs for hyperparameters (Appendix A.3) and 6×10^19 FLOPs for architecture (Appendix A.5), while LLaDA MoE v2 is trained at roughly 5×10^23 FLOPs. The single large-model run in Section 4 is not a controlled test of any individual exponent: its hyperparameters, allocation, and architecture are all read off the same fitted laws, so agreement does not validate the laws independently, and no intermediate-scale check above roughly 6×10^20 FLOPs is reported. The large-model benchmark results are practically valuable, but they should be presented as a case study consistent with the laws rather than as evidence that the fitted exponents extrapolate. I would ask the authors to either add an intermediate-scale controlled check for at least the allocation and activation-ratio dimensions or explicitly soften the abstract/conclusion claims from 'establish practical scaling laws' to a conditional statement.
minor comments (5)
  1. [Figure 4] The caption says 'Colors indicate the activated model-side budget M*(C)', but no colorbar or legend is visible in the figure; please add one.
  2. [Appendix A.5, shared-expert-ratio sweep] The statement that the activation-ratio drift in the shared-expert-ratio sweep is 'negligible' is not quantified; given that the S=33.3% optimum is a headline claim, please report the actual A values for each candidate.
  3. [§3.1 and Table 5] The text says the joint search is extended 'from 3×10^20 to 6×10^20 FLOPs' for Figure 2, but Table 5 lists the largest 3.6B-model budget as 3×10^20; please clarify the actual compute budget of the runs in Figure 2.
  4. [Table 4 and Appendix B.4] The headline 'seven of eight' comparison against SDAR Chat mixes numbers reported in prior papers with numbers from the unified evaluation; please state explicitly which entries are newly measured under the protocol in Appendix B.4.
  5. [Eqs. (2) and (3)] Please state at the equations that C is measured in non-embedding training FLOPs and that B* is a nominal token batch size, since these units are central to interpreting the coefficients.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation found: the scaling laws are empirical fits to training loss and the large-model benchmark results are external measurements, not inputs to the fits; the weak point is that LLaDA MoE v2's hyperparameters and architecture are read from the same fitted laws, making the single large run a self-referential validation rather than a circular one.

full rationale

The scaling laws in Eqs. 2 and 3 are empirical power-law fits to training loss in controlled small-scale sweeps (Appendices A.3-A.5), not outputs of LLaDA MoE v2. The benchmark results in Tables 3-5 are measured externally and are not fed back into the fits, so there is no equation that reduces to its own inputs by construction. The apparent complementarity of the model-side and data-side exponents, M* = 0.5152*C^0.475 and D* = 1.9411*C^0.525, is explicitly disclosed in Appendix A.4 as derived from D*(C)=C/M*(C); the empirical content resides in the fitted M* frontier, so this is a standard IsoFLOP identity rather than a hidden circular step. The larger concern is that LLaDA MoE v2's design (A=9.09%, G=8, S=33.3%, batch size, learning rate) is taken directly from the same fitted laws, so the single successful 30B run is a weak self-referential validation that cannot independently confirm any individual exponent; the paper itself acknowledges in the Conclusion that the scaling dimensions were varied separately and their interactions are not captured. Appendix A.5 also notes that the router contribution is ignored as 'negligible' when preserving M*(C), which is a compute-accounting risk for the architecture sweeps but is a correctness issue, not a circularity. Self-citations to prior LLaDA/SMDM work are contextual and not load-bearing for the central scaling claims. Overall, no significant circularity is present; the score of 2 reflects the minor self-referential validation design rather than a logical circle.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No new physical or theoretical entities are introduced. The central empirical laws carry five fitted constants (six if the architecture tuple is counted) and several domain assumptions about extrapolation, overtraining, and evaluation comparability. The shared-expert ratio and activation ratio are selected from small-scale sweeps rather than derived.

free parameters (6)
  • Batch-size scaling law B* = 0.374*C^0.3481 = 0.374, 0.3481
    Fit to near-optimal nominal batch sizes across 158M to 3.6B models at compute 1e18 to 3e20 FLOPs (Eq. 2).
  • Learning-rate scaling law eta* = 64.8*C^-0.2447 = 64.8, -0.2447
    Fit to near-optimal peak learning rates in the same hyperparameter sweeps (Eq. 2).
  • Model-side allocation law M* = 0.5152*C^0.475 = 0.5152, 0.475
    IsoFLOP frontier fit across compute budgets 1e17 to 1e20 FLOPs (Eq. 3).
  • Data-side allocation law D* = 1.9411*C^0.525 = 1.9411, 0.525
    Derived from D*=C/M* under the fixed-compute constraint, so its exponent is not an independent fit (Eq. 3).
  • Large-model architecture tuple (A,G,S) = 9.09%, 8, 33.3%
    Read off from small-scale architecture sweeps in Fig. 4 and not re-tuned or ablated at 30B scale.
  • Router auxiliary loss coefficients alpha_aux, alpha_z = 0.01, 0.001
    Fixed across all sweeps and the large model, following AR MoE practice; not calibrated for dLLMs.
assumptions (5)
  • standard math Power-law functional forms for B*, eta*, M*, D* are correct
    Log-log linear regressions in Eqs. 2 and 3 assume power-law scaling; no model comparison against alternatives is reported.
  • domain assumption Small-scale fits extrapolate to the 30B, roughly 5e23 FLOPs regime
    Sweeps stop at 1e20 FLOPs for hyperparameters and allocation and 6e19 FLOPs for architecture, while LLaDA MoE v2 is trained at roughly 5e23 FLOPs.
  • domain assumption Training loss averaged over the final 0.5% of FLOPs, with a 0.25% near-optimal threshold, identifies optimal hyperparameters
    Follows Bi et al. 2024; threshold choice directly influences which runs enter the regressions.
  • domain assumption Overtraining at 3D*(C) tokens in architecture sweeps represents the regime relevant for large-scale pretraining
    Used in Appendix A.5 to pick architecture recommendations; the final model is trained at roughly 4.5x D*, so this choice matters.
  • domain assumption Official-paper baseline scores are comparable to internally measured scores
    Appendix B.4 prefers official numbers; subtle evaluation differences can change benchmark deltas of several points.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models." pith.science (2026). https://pith.science/paper/27GB7ZRJ

@misc{pith2026260803457,
  author       = {Pith},
  title        = {Pith review of: LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/27GB7ZRJ}},
  note         = {Machine review of arXiv:2608.03457}
}
read the original abstract

Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling, yet the scaling behavior of Mixture-of-Experts (MoE) dLLMs remains poorly understood. We systematically characterize how optimization hyperparameters, compute allocation, and architecture scale for MoE dLLMs, identifying quantitative differences from scaling trends previously reported for AR models. Specifically, for optimization, the optimal nominal batch size grows faster, while the optimal learning rate decays more rapidly with compute. For model--data allocation, IsoFLOP analysis reveals a slight data-side tilt: the optimal token budget grows faster than activated model-side computation. For MoE architecture, larger scales increasingly favor larger expert pools at fixed activated capacity, while moderate expert granularity remains consistently effective and the preferred fraction of activated capacity assigned to shared experts remains stable across scales. Guided by these findings, we train LLaDA MoE v2, a 30B-A3B dLLM, from scratch on 23.5T tokens. With approximately 65\% as many pretraining tokens as Qwen3, LLaDA MoE v2 approaches Qwen3 on several knowledge, reasoning, and coding benchmarks. After supervised fine-tuning alone, it outperforms SDAR Chat on seven of eight reasoning and coding benchmarks and remains close to Qwen3 on several tasks. These results establish practical scaling laws and design principles for MoE dLLMs.

Figures

Figures reproduced from arXiv: 2608.03457 by the authors.

Figure 1
Figure 1. Scaling curves of nominal token batch size and learning rate with training compute. Left: nominal token batch size. Right: learning rate. Magenta dashed lines indicate our fitted scaling laws, blue dashed lines show the reference scaling laws from DeepSeek LLM (Bi et al., 2024), and shaded regions denote the empirical ranges around our fitted curves. As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Joint search over batch size and learning rate at 6×1020 FLOPs. Each cell corresponds to one training run, with color and overlaid value denoting training loss; red stars mark the fitted scaling-law predic￾tion and the best observed configuration. To test whether the fitted dLLM-specific scaling law extrapolates beyond its fitting range, we continue the joint batch-size and learning-rate runs from 3 × 1020 to 6 × 10… view at source ↗
Figure 3
Figure 3. IsoFLOP analysis for compute allocation. Left: training loss under different model– data allocations at fixed compute budgets, with stars marking the lowest-loss. Middle and right: fitted power laws for the compute-optimal non-embedding FLOPs per token and training tokens. supervised targets over a more variable set of conditioning and routing states. Additional tokens can improve coverage of both denoising targets … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: MoE architecture scaling. We evaluate how different decompositions of the activated model-side budget affect training loss across compute scales. (a): Activation Ratio A, (b): Expert Granularity G, and (c): Shared-Expert Ratio S. Colors indicate the activated model-sid…
Figure 5
Figure 5. Figure 5: Benchmark performance versus training compute. Red curves show the scaling-law￾guided LLaDA MoE v2 30B-A3B model evaluated with varying training-token budgets, and the black diamond denotes the LLaDA MoE 7B-A1B model trained without scaling-law guidance. 4.2 SUPERVISED…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

90 extracted references · 17 canonical work pages

  1. [1]

    Frontiers of Computer Science , volume=

    A survey of large language models , author=. Frontiers of Computer Science , volume=. 2026 , publisher=

    2026

  2. [2]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    arXiv

  3. [3]

    arXiv preprint arXiv:2309.16609 , year=

    Qwen technical report , author=. arXiv preprint arXiv:2309.16609 , year=

    arXiv

  4. [4]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  5. [5]

    arXiv preprint arXiv:2401.02954 , year=

    Deepseek llm: Scaling open-source language models with longtermism , author=. arXiv preprint arXiv:2401.02954 , year=

    arXiv

  6. [6]

    arXiv preprint arXiv:2405.04434 , year=

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model , author=. arXiv preprint arXiv:2405.04434 , year=

    arXiv

  7. [7]

    arXiv preprint arXiv:2412.19437 , year=

    Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=

    arXiv

  8. [8]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    arXiv

Show all 90 references
  1. [9]

    arXiv preprint arXiv:2507.06261 , year=

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    arXiv

  2. [10]

    Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

  3. [11]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    arXiv

  4. [12]

    arXiv preprint arXiv:1712.00409 , year=

    Deep learning scaling is predictable, empirically , author=. arXiv preprint arXiv:1712.00409 , year=

    arXiv

  5. [13]

    arXiv preprint arXiv:2001.08361 , year=

    Scaling laws for neural language models , author=. arXiv preprint arXiv:2001.08361 , year=

    2001 arXiv

  6. [14]

    arXiv preprint arXiv:2203.15556 , year=

    Training compute-optimal large language models , author=. arXiv preprint arXiv:2203.15556 , year=

    arXiv

  7. [15]

    International conference on machine learning , pages=

    Unified scaling laws for routed language models , author=. International conference on machine learning , pages=. 2022 , organization=

    2022

  8. [16]

    International Conference on Learning Representations , volume=

    Language models scale reliably with over-training and on downstream tasks , author=. International Conference on Learning Representations , volume=

  9. [17]

    arXiv preprint arXiv:2501.12370 , year=

    Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models , author=. arXiv preprint arXiv:2501.12370 , year=

    arXiv

  10. [18]

    arXiv preprint arXiv:2502.05172 , year=

    Joint moe scaling laws: Mixture of experts can be memory efficient , author=. arXiv preprint arXiv:2502.05172 , year=

    arXiv

  11. [19]

    arXiv preprint arXiv:2503.04715 , year=

    Predictable Scale: Part I, Step Law--Optimal Hyperparameter Scaling Law in Large Language Model Pretraining , author=. arXiv preprint arXiv:2503.04715 , year=

    arXiv

  12. [20]

    International Conference on Learning Representations , volume=

    Towards greater leverage: Scaling laws for efficient mixture-of-experts language models , author=. International Conference on Learning Representations , volume=

  13. [21]

    2018 , publisher=

    Improving language understanding by generative pre-training , author=. 2018 , publisher=

    2018

  14. [22]

    OpenAI blog , volume=

    Language models are unsupervised multitask learners , author=. OpenAI blog , volume=

  15. [23]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  16. [24]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  17. [25]

    Advances in Neural Information Processing Systems , volume=

    A continuous time framework for discrete denoising models , author=. Advances in Neural Information Processing Systems , volume=

  18. [26]

    Advances in neural information processing systems , volume=

    Structured denoising diffusion models in discrete state-spaces , author=. Advances in neural information processing systems , volume=

  19. [27]

    Advances in Neural Information Processing Systems , volume=

    Likelihood-based diffusion language models , author=. Advances in Neural Information Processing Systems , volume=

  20. [28]

    Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    Diffusionbert: Improving generative masked language models with diffusion models , author=. Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

  21. [29]

    arXiv preprint arXiv:2208.04202 , year=

    Analog bits: Generating discrete data using diffusion models with self-conditioning , author=. arXiv preprint arXiv:2208.04202 , year=

    arXiv

  22. [30]

    arXiv preprint arXiv:2404.15766 , year=

    Unifying bayesian flow networks and diffusion models through stochastic differential equations , author=. arXiv preprint arXiv:2404.15766 , year=

    arXiv

  23. [31]

    arXiv preprint arXiv:2310.16834 , year=

    Discrete diffusion modeling by estimating the ratios of the data distribution , author=. arXiv preprint arXiv:2310.16834 , year=

    arXiv

  24. [32]

    Advances in neural information processing systems , volume=

    Simplified and generalized masked diffusion for discrete data , author=. Advances in neural information processing systems , volume=

  25. [33]

    Advances in Neural Information Processing Systems , volume=

    Simple and effective masked diffusion language models , author=. Advances in Neural Information Processing Systems , volume=

  26. [34]

    International Conference on Learning Representations , volume=

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data , author=. International Conference on Learning Representations , volume=

  27. [35]

    International Conference on Learning Representations , volume=

    Scaling up masked diffusion models on text , author=. International Conference on Learning Representations , volume=

  28. [36]

    arXiv preprint arXiv:2510.03280 , year=

    Training optimal large diffusion language models , author=. arXiv preprint arXiv:2510.03280 , year=

  29. [37]

    International Conference on Learning Representations , volume=

    Scaling behavior of discrete diffusion language models , author=. International Conference on Learning Representations , volume=

  30. [38]

    International Conference on Learning Representations , volume=

    Block diffusion: Interpolating between autoregressive and diffusion language models , author=. International Conference on Learning Representations , volume=

  31. [39]

    Advances in Neural Information Processing Systems , volume=

    Large language diffusion models , author=. Advances in Neural Information Processing Systems , volume=

  32. [40]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Llada 1.5: Variance-reduced preference optimization for large language diffusion models , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  33. [41]

    arXiv preprint arXiv:2606.25331 , year=

    Improved Large Language Diffusion Models , author=. arXiv preprint arXiv:2606.25331 , year=

    arXiv

  34. [42]

    International Conference on Learning Representations , volume=

    Scaling diffusion language models via adaptation from autoregressive models , author=. International Conference on Learning Representations , volume=

  35. [43]

    Dream 7B , url =

    Ye, Jiacheng and Xie, Zhihui and Zheng, Lin and Gao, Jiahui and Wu, Zirui and Jiang, Xin and Li, Zhenguo and Kong, Lingpeng , year =. Dream 7B , url =

  36. [44]

    arXiv preprint arXiv:2509.24389 , year=

    LLaDA-MoE: A sparse MoE diffusion language model , author=. arXiv preprint arXiv:2509.24389 , year=

  37. [45]

    arXiv preprint arXiv:2508.02193 , year=

    Seed diffusion: A large-scale diffusion language model with high-speed inference , author=. arXiv preprint arXiv:2508.02193 , year=

    arXiv

  38. [46]

    arXiv preprint arXiv:2506.17298 , year=

    Mercury: Ultra-fast language models based on diffusion , author=. arXiv preprint arXiv:2506.17298 , year=

    arXiv

  39. [47]

    arXiv preprint arXiv:2508.10875 , year=

    A survey on diffusion language models , author=. arXiv preprint arXiv:2508.10875 , year=

    arXiv

  40. [48]

    International Conference on Learning Representations , volume=

    Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling , author=. International Conference on Learning Representations , volume=

  41. [49]

    International Conference on Learning Representations , volume=

    Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding , author=. International Conference on Learning Representations , volume=

  42. [50]

    arXiv e-prints , pages=

    Accelerating diffusion large language models with slowfast: The three golden principles , author=. arXiv e-prints , pages=

  43. [51]

    International Conference on Learning Representations , volume=

    Refusion: A diffusion large language model with parallel autoregressive decoding , author=. International Conference on Learning Representations , volume=

  44. [52]

    arXiv preprint arXiv:2602.15014 , year=

    Scaling beyond masked diffusion language models , author=. arXiv preprint arXiv:2602.15014 , year=

  45. [53]

    0: Scaling up diffusion language models to 100b , author=

    Llada2. 0: Scaling up diffusion language models to 100b , author=. arXiv preprint arXiv:2512.15745 , year=

    arXiv

  46. [54]

    arXiv preprint arXiv:2511.03276 , year=

    Diffusion language models are super data learners , author=. arXiv preprint arXiv:2511.03276 , year=

  47. [55]

    Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    2026

  48. [56]

    arXiv preprint arXiv:2605.30876 , year=

    dMoE: dLLMs with Learnable Block Experts , author=. arXiv preprint arXiv:2605.30876 , year=

    arXiv

  49. [57]

    arXiv preprint arXiv:2604.01622 , year=

    Expert-choice routing enables adaptive computation in diffusion language models , author=. arXiv preprint arXiv:2604.01622 , year=

    arXiv

  50. [58]

    arXiv preprint arXiv:2602.06036 , year=

    Dflash: Block diffusion for flash speculative decoding , author=. arXiv preprint arXiv:2602.06036 , year=

    arXiv

  51. [59]

    arXiv preprint arXiv:2607.05147 , year=

    DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation , author=. arXiv preprint arXiv:2607.05147 , year=

    arXiv

  52. [60]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  53. [61]

    arXiv preprint arXiv:1711.05101 , year=

    Decoupled weight decay regularization , author=. arXiv preprint arXiv:1711.05101 , year=

    arXiv

  54. [62]

    arXiv preprint arXiv:2002.05202 , year=

    Glu variants improve transformer , author=. arXiv preprint arXiv:2002.05202 , year=

    2002 arXiv

  55. [63]

    Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints , author=. Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    2023

  56. [64]

    Neurocomputing , volume=

    Roformer: Enhanced transformer with rotary position embedding , author=. Neurocomputing , volume=. 2024 , publisher=

    2024

  57. [65]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    Effective long-context scaling of foundation models , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2024

  58. [66]

    International Conference on Learning Representations , volume=

    Mixture-of-experts meets instruction tuning: A winning combination for large language models , author=. International Conference on Learning Representations , volume=

  59. [67]

    arXiv preprint arXiv:2402.07871 , year=

    Scaling laws for fine-grained mixture of experts , author=. arXiv preprint arXiv:2402.07871 , year=

    arXiv

  60. [68]

    arXiv preprint arXiv:2401.04088 , year=

    Mixtral of experts , author=. arXiv preprint arXiv:2401.04088 , year=

    arXiv

  61. [69]

    arXiv preprint arXiv:1701.06538 , year=

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer , author=. arXiv preprint arXiv:1701.06538 , year=

    arXiv

  62. [70]

    arXiv preprint arXiv:2006.16668 , year=

    Gshard: Scaling giant models with conditional computation and automatic sharding , author=. arXiv preprint arXiv:2006.16668 , year=

    2006 arXiv

  63. [71]

    Journal of Machine Learning Research , volume=

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity , author=. Journal of Machine Learning Research , volume=

  64. [72]

    Neural computation , volume=

    Adaptive mixtures of local experts , author=. Neural computation , volume=. 1991 , publisher=

    1991

  65. [73]

    arXiv preprint arXiv:2202.08906 , year=

    St-moe: Designing stable and transferable sparse expert models , author=. arXiv preprint arXiv:2202.08906 , year=

    arXiv

  66. [74]

    arXiv preprint arXiv:2502.16982 , year=

    Muon is scalable for llm training , author=. arXiv preprint arXiv:2502.16982 , year=

    arXiv

  67. [75]

    International conference on machine learning , pages=

    Glam: Efficient scaling of language models with mixture-of-experts , author=. International conference on machine learning , pages=. 2022 , organization=

    2022

  68. [76]

    arXiv preprint arXiv:2009.03300 , year=

    Measuring massive multitask language understanding , author=. arXiv preprint arXiv:2009.03300 , year=

    2009 arXiv

  69. [77]

    Advances in Neural Information Processing Systems , volume=

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark , author=. Advances in Neural Information Processing Systems , volume=

  70. [78]

    Advances in neural information processing systems , volume=

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models , author=. Advances in neural information processing systems , volume=

  71. [79]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    Cmmlu: Measuring massive multitask language understanding in chinese , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024

  72. [80]

    Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

    Hellaswag: Can a machine really finish your sentence? , author=. Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

  73. [81]

    International Conference on Learning Representations , volume=

    Kor-bench: Benchmarking language models on knowledge-orthogonal reasoning tasks , author=. International Conference on Learning Representations , volume=

  74. [82]

    arXiv preprint arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    arXiv

  75. [83]

    arXiv preprint arXiv:2103.03874 , year=

    Measuring mathematical problem solving with the math dataset , author=. arXiv preprint arXiv:2103.03874 , year=

    arXiv

  76. [84]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  77. [85]

    arXiv preprint arXiv:2401.03065 , year=

    Cruxeval: A benchmark for code reasoning, understanding and execution , author=. arXiv preprint arXiv:2401.03065 , year=

    arXiv

  78. [86]

    arXiv preprint arXiv:2108.07732 , year=

    Program synthesis with large language models , author=. arXiv preprint arXiv:2108.07732 , year=

    arXiv

  79. [87]

    arXiv preprint arXiv:2208.08227 , year=

    Multipl-e: A scalable and extensible approach to benchmarking neural code generation , author=. arXiv preprint arXiv:2208.08227 , year=

    arXiv

  80. [88]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

    arXiv

  81. [89]

    International Conference on Learning Representations , volume=

    Livecodebench: Holistic and contamination free evaluation of large language models for code , author=. International Conference on Learning Representations , volume=

  82. [90]

    International Conference on Learning Representations , volume=

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions , author=. International Conference on Learning Representations , volume=

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.