Pith. sign in

REVIEW 3 major objections 4 minor 67 references

Scaling Laws for Upcycling Mixture-of-Experts Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that upcycled MoE language models follow a two-stage power law with a positive interaction term, which makes upcycling beat from-scratch training only below a model-size-dependent token threshold.

desk verdict A genuinely useful empirical scaling law for MoE upcycling, with honest limitations, but the headline threshold rests on an unexplained order-of-magnitude discrepancy in the fitted interaction exponent. read the letter →

arxiv 2502.03009 v2 pith:DXBWIRUN submitted 2025-02-05 cs.LG cs.CL

Seng Pei Liew , Takuya Kato , Sho Takase This is my paper · ORCID
classification cs.LGcs.CL
keywords mixture-of-expertsupcyclingscalinglawslanguagemodelpretrainingpowerlawcompute-optimaltrainingsparsity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish a predictive scaling law for upcycling a pretrained dense language model into a mixture-of-experts (MoE) model. The proposed law is $L = A D_1^{-\alpha_1} D_2^{-(\alpha_2+\alpha_3 \log D_1)} + B N_1^{-\beta} + E$, where $D_1$ is the dense pretraining budget, $D_2$ the continued-pretraining budget of the upcycled MoE, and $N_1$ the dense model size. The novel, load-bearing piece is the positive interaction exponent $\alpha_3$: the more tokens already sunk into the dense model, the slower the upcycled MoE improves with additional tokens. If this law is right, it resolves contradictory reports about upcycling by showing that upcycling is more efficient than from-scratch MoE training only when $D_2$ stays below a threshold $D^*$ that shrinks as the model grows. Practitioners can use the fitted law to decide when reusing a checkpoint saves compute and when it does not.

What carries the argument

The load-bearing object is Equation (12), the joint scaling law $L = A D_1^{-\alpha_1} D_2^{-(\alpha_2+\alpha_3 \log D_1)} + B N_1^{-\beta} + E$. It is a multiplicative power law in the two token budgets whose $D_2$ exponent is modulated by $\log D_1$, so the two budgets interact rather than contributing additively. The paper arrives at this form by imposing three requirements: power-law scaling in $D_2$ for fixed $D_1$; reduction to the dense Chinchilla law $A D_1^{-\alpha_1} + B N_1^{-\beta} + E$ as $D_2 \to 0$; and the empirical observation that the fitted $D_2$ exponent falls linearly in $\log D_1$ (Equation 10), which turns the single-variable power law into a bilinear form in $\log D_1$ and $\log D_2$. This interaction term carries the argument because it is what makes upcycling advantages shrink with sunk cost and produces the threshold $D^*$.

What would settle it

At the 8x1B scale the paper predicts $D^* = 4$B tokens: train a 1B dense model on 4B tokens, upcycle it, and in parallel train an 8x1B MoE from scratch for 4B tokens; the law requires the two final losses to match, and requires from-scratch to win for any $D_2 > D^*$. Running that comparison, or the analogous one at 7B scale with $D_1$ far above the fitted range, would falsify the scaling law if the crossover token count is off.

Watch

Extended reading notes

Core claim

The central discovery is that the upcycled MoE loss has a multiplicative power-law structure with an interaction between the two training budgets, not just two additive terms. The paper states it as $L(D_1,D_2,N_1) = A D_1^{-\alpha_1} D_2^{-(\alpha_2+\alpha_3 \log D_1)} + B N_1^{-\beta} + E$, fitted at 8-expert top-2 Mixtral-like architectures with positive exponents ($\alpha_1 \approx 0.043$, $\alpha_2 \approx 0.085$, $\alpha_3 \approx 7.98\times 10^{-4}$, $\beta \approx 0.112$). The interaction term means the effective exponent of $D_2$ is $\alpha_2 - \alpha_3 \log D_1$, so a larger dense pretraining budget lowers the starting loss but also slows later progress. From this the paper derives a threshold $D^* \simeq 4 (N_1/10^9)^{-0.7+0.04\log(N_1/10^9)}$ billion tokens separating regimes where upcycling beats from-scratch MoE training, shows that from-scratch training is compute-optimal when no pretrained checkpoint exists, and obtains $D_2 \propto N_1^{1.8}$ for compute-optimal upcycling of an existing checkpoint.

Load-bearing premise

The argument stands on the assumption that the measured slowdown in upcycling gains keeps following the same straight-line trend as the dense pretraining budget grows, all the way out to 7B, 13B, and 70B models trained on roughly two trillion tokens; if that trend bends or changes with model size, the predicted threshold and compute guidance collapse.

Editorial extensions

If this is right

  • Upcycling a dense checkpoint into an 8-expert top-2 MoE is more token-efficient than from-scratch MoE training only while the additional upcycling budget $D_2$ stays below $D^*$; above $D^*$, from-scratch training wins.
  • The threshold falls with dense model size, so for larger models upcycling pays off only for very small continued-training budgets: the paper's formula gives $D^* \simeq 4 (N_1/10^9)^{-0.7+0.04\log(N_1/10^9)}$ billion tokens.
  • If no pretrained checkpoint is available, compute-optimal dense or from-scratch MoE training outperforms the two-stage dense-to-upcycled pipeline under the fitted scaling law.
  • When a pretrained dense checkpoint already exists, compute-optimal upcycling obeys $D_2 \propto N_1^{1.8}$, so larger dense models need nearly quadratically more upcycling tokens to remain efficient.
  • Within the explored ranges, increasing sparsity and active parameters improves the upcycled MoE's loss without a noticeable performance trade-off.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same interaction structure would apply to other two-stage training regimes, which the paper notes but does not develop: additive scaling laws for transfer learning and fine-tuning would systematically over-value the pretraining budget at large $D_1$.
  • The linear dependence on $\log D_1$ predicts a crossover at which, for a fixed $D_2$, more dense pretraining stops helping; measuring losses at $D_1$ well beyond the fitted range would test whether the linear form bends before that crossover.
  • The appendix's dataset comparison hints that the interaction exponent depends on task difficulty, with code saturating quickly and Japanese text saturating slowly, so fitting $\alpha_3$ per dataset and checking whether $D^*$ moves is a natural next experiment.
  • A practical decision rule follows from the threshold: fix the MoE architecture, estimate $D^*$, and only upcycle when the planned continued-pretraining budget is below it; otherwise train from scratch.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies upcycling a pretrained dense language model into a mixture-of-experts (MoE) model by duplicating MLP weights and continuing training. It fits an empirical scaling law L(D1,D2,N1) = A D1^{-alpha1} D2^{-(alpha2 + alpha3 log D1)} + B N1^{-beta} + E, with a positive interaction exponent alpha3, and uses it to derive a threshold D* below which upcycling beats from-scratch MoE training (Eq. 2) and to give compute-optimal allocation guidance (Eq. 14). The experiments cover dense models from 15M to 1B parameters, several MoE configurations, multiple datasets, and the paper releases code and loss data.

Significance. If the proposed form is correct, this would be one of the first systematic scaling laws for upcycling and would give practitioners a concrete decision rule with a nontrivial interaction between the dense pretraining budget and the upcycled training budget. The paper has real empirical support: the multiplicative form with interaction wins leave-one-out comparisons across four MoE architectures in Table 1, the joint law extrapolates to a held-out 8x1B model with RMS error 0.015 in Figure 6, and the released code and data make the results reproducible. The main caveat is quantitative: the interaction exponent alpha3, which drives the headline threshold and compute-optimal conclusions, is fit-dependent by an order of magnitude across the paper's own tables, and no uncertainty intervals are reported. Because Section 7 explicitly states that the interaction term is theoretically unexplained, the numerical robustness of alpha3 is load-bearing for the paper's conclusions.

major comments (3)
  1. [Section 4.1/Table 8 vs Section 5/Table 2] The fitted interaction exponent alpha3 differs by an order of magnitude between the dataset-size fits and the joint law used for the headline applications. Table 8 reports alpha3 in the range 0.008-0.01 for 8x0.1B upcycled MoEs, while Table 2 reports alpha3 = 7.98e-4 for the joint law. Since alpha3 is the term that makes the effective D2 exponent alpha_eff = alpha2 - alpha3 log D1 decrease with D1 and directly enters the threshold D* (Eqs. 2 and 13) and the compute-optimal exponent (Eq. 14), the paper needs either to reconcile these two fits or to provide uncertainty intervals showing that they are statistically compatible. As written, the Section 4.1 scaling law does not transfer to the exact joint-law setting used for Eqs. 2 and 14, and no explanation is given for the discrepancy.
  2. [Section 5.2 / Eq. 14] The application to Llama2 models (7B, 13B, 70B, D1 about 2T tokens) extrapolates Eq. 12 far outside the fitted range: the joint law is fit with N1 at most 1B and D1 values in the few-billion-token range. Because alpha_eff appears in the denominator of the exponent beta/alpha_eff in Eq. 14, the claim D2 ~ N1^1.8 is highly sensitive to the value of alpha3 at large log D1. With no confidence interval on alpha3, the reported near-quadratic relation is not robust; a modest perturbation of alpha3 changes the exponent substantially. The held-out validation at 8x1B does not cover this regime.
  3. [Section 5.1 / Eq. 13] The threshold D* is the root of an equality between two fitted scaling laws and is therefore an algebraic consequence of the fit, not an independent measurement. The manuscript presents D* as a headline prediction, and Figure 1 extends it to model sizes beyond the fitted range. At minimum, the paper should state clearly that D* inherits all the uncertainty of both fitted laws, and should provide a sensitivity analysis with respect to the fitted exponents, in particular alpha3.
minor comments (4)
  1. [Section 4.1 / Eq. 7] Equation 7 writes the prefactor as A independent of D1, but later in the same section the prefactor is fitted as a function of D1 (Eq. 16 and Figure 12); please make the D1-dependence of the prefactor explicit in Eqs. 7-9.
  2. [Section 4.1 / Eq. 10] Equation 10 uses E' for the intercept although E denotes the irreducible loss elsewhere; this overloading is confusing and should be relabeled.
  3. [Appendix C.1] The text says 'Figure 8 shows the fitting' but the fitted exponents appear in Table 8; the cross-reference should be corrected.
  4. [Section 5.2] The compute cost is written as C2 = 6 N2 D2, but it is not stated whether the FLOPs budget C in Eq. 14 refers only to the upcycling stage or to the sum of dense and upcycled training; please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the scaling laws are explicitly fitted empirical models, and the threshold and compute-optimal relations are openly derived consequences of those fits rather than independently predicted data.

full rationale

Section 2.2 introduces the loss model as an empirical ansatz: "All parameters are to be fitted with experimental observations." Equation 12 is therefore a fitted regression surface, not a first-principles theorem. The interaction term α3 is not smuggled in: Table 1 compares multiplicative and additive forms with and without interaction by leave-one-out RMS, and Section 4.1 together with Appendix C shows that the multiplicative dependence arises from independent empirical fits (Equations 10 and 16). The later quantities are consequences: D* is defined in Equation 13 as the root of two fitted laws, Equation 2 is derived as its approximate analytic solution, and Equation 14 is derived by differentiating the fitted joint law under a FLOP constraint in Appendix F.3. These are legitimate applications of a fitted model, and the paper does not relabel regression output as an independent measurement. Held-out validation (RMS 0.015 in Section 5) and cross-dataset generalization checks (Appendix B) are independent checks rather than circular reuse of the fitted data. The citations to prior scaling work supply a standard ansatz, and the cited Mixtral and Sarashina2 models are used for architecture choices, not to prove the law; there is no load-bearing self-citation or imported uniqueness theorem. The paper honestly states in Section 7 that the interaction term "remains theoretically unexplained," and the numerical discrepancy in α3 between Table 2 and Table 8 is a correctness or extrapolation concern, not a definitional reduction: both values are fitted from data and neither is defined in terms of the threshold or the compute-optimal exponent. Because every claimed prediction is either a fitted functional form or an explicit algebraic consequence of that form, no circular step can be exhibited.

Assumptions & free parameters 9 free parameters · 6 assumptions · 0 invented entities

The paper's central claim rests on nine fitted parameters and six assumptions. None of the assumptions is unreasonable for an empirical scaling-law study, but the interaction term and the extrapolated threshold depend especially on the log-linear exponent model and on the universal-E assumption. No new physical entities are introduced.

free parameters (9)
  • A (upcycled prefactor) = 16.3
    Fitted from cross-entropy losses over dataset and model size grid (Table 2); sets the overall scale of the D1/D2 terms in Eq. 12.
  • B (model-size prefactor) = 8.53
    Fitted from N1 data (Table 2); enters Eq. 12 and therefore the threshold D*.
  • alpha_1 (dense-token exponent) = 0.043
    Dense pretraining token exponent for the upcycled law, fitted (Table 2).
  • alpha_2 (upcycled-token exponent) = 0.085
    Upcycled training token exponent, fitted (Table 2); larger than alpha_1 is part of the argument that upcycling has a head start.
  • alpha_3 (interaction exponent) = 7.98e-4 in Table 2; 0.008-0.01 in Table 8
    The novel interaction exponent, fitted rather than derived, and the exact value appears to depend on fitting setup and units, which is not discussed.
  • beta (upcycled N1 exponent) = 0.112
    Model-size exponent in Eq. 12; fitted (Table 2).
  • E (irreducible loss) = 0.165
    Fitted jointly from dense and MoE laws, then fixed for the upcycled fit; assumes a universal entropy floor across all three training regimes.
  • beta_1 (sparsity exponent) = 1.87
    Fitted from the P,N2 grid in Eq. 11; supports the claim that more sparsity improves loss, with no error bars.
  • beta_2 (active-parameter exponent) = 0.34
    Fitted from the P,N2 grid in Eq. 11; supports the claim that more active parameters improve loss in the tested range.
assumptions (6)
  • domain assumption Power-law ansatz L = A X^-alpha + E for loss versus dataset and model size.
    Invoked in Section 2.2 and borrowed from prior scaling-law literature (Hoffmann et al. 2022); it is an empirical convention rather than a proven law.
  • domain assumption Function-preserving initialization makes the D2 -> 0 limit equal to the dense scaling law.
    Used as Requirement 2 in Section 4.1; it holds approximately because expert weights are duplicated from the dense model and routing preserves the dense output at initialization.
  • ad hoc to paper The multiplicative form with a log D1 interaction is the correct interpolation between the dense and upcycled limits.
    Selected by leave-one-out RMS comparison in Table 1; the additive form and the no-interaction forms fit worse, but the form itself is not derived from a mechanism.
  • ad hoc to paper The scaling exponent alpha(D1) is linear in log D1.
    Modeled in Eq. 10, Section 4.1, from a small number of fitted exponents; this log-linear assumption directly produces the interaction term.
  • domain assumption The irreducible loss E is universal across dense, MoE, and upcycled training.
    Imposed in the fitting procedure in Section 5: E is estimated from dense and MoE laws, then fixed when fitting the upcycled law.
  • ad hoc to paper Fitted scaling-law parameters remain valid when extrapolated to 7B-70B models and roughly 2T dense tokens.
    Assumed in Section 5.2 when applying Eq. 12 and Eq. 14 to Llama2-scale models; this is an unvalidated extrapolation far beyond the fit range.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Laws for Upcycling Mixture-of-Experts Language Models." pith.science (2026). https://pith.science/paper/DXBWIRUN

@misc{pith2026250203009,
  author       = {Pith},
  title        = {Pith review of: Scaling Laws for Upcycling Mixture-of-Experts Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DXBWIRUN}},
  note         = {Machine review of arXiv:2502.03009}
}
read the original abstract

Pretraining large language models (LLMs) is resource-intensive, often requiring months of training time even with high-end GPU clusters. There are two approaches of mitigating such computational demands: reusing smaller models to train larger ones (upcycling), and training computationally efficient models like mixture-of-experts (MoE). In this paper, we study the upcycling of LLMs to MoE models, of which the scaling behavior remains underexplored. Through extensive experiments, we identify empirical scaling laws that describe how performance depends on dataset size and model configuration. Particularly, we show that, while scaling these factors improves performance, there is a novel interaction term between the dense and upcycled training dataset that limits the efficiency of upcycling at large computational budgets. Based on these findings, we provide guidance to scale upcycling, and establish conditions under which upcycling outperforms from-scratch trainings within budget constraints.

Figures

Figures reproduced from arXiv: 2502.03009 by the authors.

Figure 1
Figure 1. Left: Upcycling improves with sparsity and the number of active parameters. We find that upcycling to MoE which is sparser and has more active parameters improves performance. The z-axis shows the value of cross-entropy loss. See Section 4.2 for details. Right: Efficiency of upcycling diminishes with sunk cost and model size. For all additional token budgets for upcycling a Mixtral-like MoE above the curve(s), train… view at source ↗
Figure 2
Figure 2. Upcycling and factors affecting MoE’s performance. Upcycling involves initiating the weights of the MoE (activating nTopK experts per token) by reusing the weights (duplicating the weights of MLPs nexpert times) of an existing dense transformer of size N1 that has been trained for D1 tokens. The (upcycled) MoE is further trained for D2 tokens. Language modeling performance improves when scaling these factors. We stu… view at source ↗
Figure 3
Figure 3. Loss curves of upcycling. Intermediate test losses of the 8x0.1B MoE (2 experts activated per token) trained for a variety of total number of tokens, D2, when upcycled from a dense model pretrained with various numbers of training tokens (D1) in B. 4. Scaling Laws Our ultimate goal is to understand the performance of upcy￾cling with respect to various factors illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Top: D2 has power-law scaling. We show scaling behavior of upcycled training tokens (D2) for different values of dense tokens (D1). Bottom: Interaction term explains decreasing exponents. The fitted exponents in the upper plots are used to fit Equation 10 as a function…
Figure 5
Figure 5. Figure 5: Fits of scaling law for model configuration. To do this, we define sparsity, P := Ntotal/N2, and con￾sider the dependency of the performance on sparsity and active parameter N2, of which the variables capture scaling behavior with respect to model parameters and archit…
Figure 6
Figure 6. Figure 6: Fits of the joint upcycling scaling law. A B α/α1 α2 α3 β Dense 8.83 12.3 0.088 − − 0.116 MoE 32.0 7.05 0.161 − − 0.080 Upcycled 16.3 8.53 0.043 0.085 7.98e-4 0.112 [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Comparing WSD and cosine schedules of learning rate. Left: we see that different schedules cause little differences between the losses, for both dense and MoE training. Right: the learning rate schedules in use are shown. A.5. Training configuration The common setup of…
Figure 8
Figure 8. Figure 8: Ablation of LR when upcycling an 8x0.1B MoE. We compare the performance of upcycled training (from a dense model trained for 2B tokens) using constant LR (2 × 10−4 , LR at the end of dense training), LR used for dense training, and LR used for MoE training. We find tha…
Figure 9
Figure 9. Figure 9: Ablating auxiliary coefficients. Left: Cross-entropy losses, where it can be seen that auxiliary coefficient of 1 performs worst. Right: Load-balancing losses, where the larger the coefficient is, the smaller the load-balancing loss becomes. Setting the coefficient to …
Figure 10
Figure 10. Figure 10: Ablation of data repetition when upcycling an 8x0.1B MoE. We do not observe notable difference in the loss. The training setup follows [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Scaling behavior generalizes across datasets used. Left: Japanese dataset. Right: Code dataset. C. More Results on the Scaling Law for Dataset Sizes C.1. Fitting across architectures [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Top: Fitting plots same as those in [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Scaling behavior with respect to model configuration. Shown are plots of the cross-entropy loss with respect to active parameter N2 fixing sparsity P. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Upcycled model has power-law behavior with respect to N1 [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: Token budget of from-scratch MoE training and when it catches up with upcycled MoE’s performance. We compare loss-versus-token plots of from-scratch and upcycled MoE trainings at various dense training budgets (sunk costs) and model sizes. Upcycling is considered to b…
Figure 16
Figure 16. Figure 16: Compute-optimal training. Upcycled training performs worse even when D1, D2, N1 are allocated optimally, compared to compute-optimal training of dense or MoE models. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 22 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Explaining neural scaling laws

    Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121 0 (27): 0 e2311878121, 2024

    work page 2024

  4. [4]

    Estimating or propagating gradients through stochastic neurons for conditional computation

    Bengio, Y., L \'e onard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013

    arXiv 2013

  5. [5]

    Deepseek llm: Scaling open-source language models with longtermism

    Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024

    arXiv 2024

  6. [6]

    G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M

    Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. pp.\ 2397--2430. PMLR, 2023

    work page 2023

  7. [7]

    Piqa: Reasoning about physical commonsense in natural language

    Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 7432--7439, 2020

    2020

  8. [8]

    A dynamical model of neural scaling laws

    Bordelon, B., Atanasov, A., and Pehlevan, C. A dynamical model of neural scaling laws. In Proceedings of the 41st International Conference on Machine Learning, pp.\ 4345--4382, 2024

    work page 2024

Show all 67 references
  1. [9]

    A survey on mixture of experts

    Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., and Huang, J. A survey on mixture of experts. arXiv preprint arXiv:2407.06204, 2024

    2024 arXiv

  2. [10]

    Net2net: Accelerating learning via knowledge transfer

    Chen, T., Goodfellow, I., and Shlens, J. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641, 2015

    2015 arXiv

  3. [11]

    W., Sutton, C., Gehrmann, S., et al

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  4. [12]

    Unified scaling laws for routed language models

    Clark, A., de Las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., Borgeaud, S., et al. Unified scaling laws for routed language models. pp.\ 4057--4086. PMLR, 2022

    2022

  5. [13]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  6. [14]

    Redpajama: an open dataset for training large language models, October 2023

    Computer, T. Redpajama: an open dataset for training large language models, October 2023. URL https://github.com/togethercomputer/RedPajama-Data

    2023

  7. [15]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models

    Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024

    2024 arXiv

  8. [16]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Dao, T., Fu, D., Ermon, S., Rudra, A., and R \'e , C. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 0 16344--16359, 2022

    2022

  9. [17]

    Stacking your transformers: A closer look at model growth for efficient llm pre-training

    Du, W., Luo, T., Qiu, Z., Huang, Z., Shen, Y., Cheng, R., Guo, Y., and Fu, J. Stacking your transformers: A closer look at model growth for efficient llm pre-training. arXiv preprint arXiv:2405.15319, 2024

    2024 arXiv

  10. [18]

    Learning factored representations in a deep mixture of experts

    Eigen, D., Ranzato, M., and Sutskever, I. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013

    2013 arXiv

  11. [19]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 0 (120): 0 1--39, 2022

    2022

  12. [20]

    A framework for few-shot language model evaluation, 07 2024

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2024

  13. [21]

    Scaling laws and compute-optimal training beyond fixed training durations

    H \"a gele, A., Bakouch, E., Kosson, A., Von Werra, L., Jaggi, M., et al. Scaling laws and compute-optimal training beyond fixed training durations. Advances in Neural Information Processing Systems, 37: 0 76232--76264, 2024

    2024

  14. [22]

    Upcycling large language models into mixture of experts

    He, E., Khattar, A., Prenger, R., Korthikanti, V., Yan, Z., Liu, T., Fan, S., Aithal, A., Shoeybi, M., and Catanzaro, B. Upcycling large language models into mixture of experts. arXiv preprint arXiv:2410.07524, 2024

    2024 arXiv

  15. [23]

    B., Dhariwal, P., Gray, S., et al

    Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020

    2010 arXiv

  16. [24]

    Scaling laws for transfer

    Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021

    2021 arXiv

  17. [25]

    Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017

    2017 arXiv

  18. [26]

    Beyond human-level accuracy: Computational challenges in deep learning

    Hestness, J., Ardalani, N., and Diamos, G. Beyond human-level accuracy: Computational challenges in deep learning. pp.\ 1--14, 2019

    2019

  19. [27]

    A., Welbl, J., Clark, A., et al

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. Proceedings of the 36th International Conference on Neural Information Processing ...

    2022

  20. [28]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024

    2024 arXiv

  21. [29]

    Learning curve theory

    Hutter, M. Learning curve theory. arXiv preprint arXiv:2102.04074, 2021

    2021 arXiv

  22. [30]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  23. [31]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  24. [32]

    M., Hughes, S., Wolf, T., Bahdanau, D., et al

    Kocetkov, D., Li, R., Jia, L., Mou, C., Jernite, Y., Mitchell, M., Ferrandis, C. M., Hughes, S., Wolf, T., Bahdanau, D., et al. The stack: 3 tb of permissively licensed source code. Transactions on Machine Learning Research

  25. [33]

    R., Mustafa, B., Ainslie, J., Tay, Y., Dehghani, M., and Houlsby, N

    Komatsuzaki, A., Puigcerver, J., Lee-Thorp, J., Ruiz, C. R., Mustafa, B., Ainslie, J., Tay, Y., Dehghani, M., and Houlsby, N. Sparse upcycling: Training mixture-of-experts from dense checkpoints

  26. [34]

    A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B

    Korthikanti, V. A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5: 0 341--353, 2023

    2023

  27. [35]

    Scaling laws for fine-grained mixture of experts

    Krajewski, J., Ludziejewski, J., Adamczewski, K., Pi \'o ro, M., Krutul, M., Antoniak, S., Ciebiera, K., Kr \'o l, K., Odrzyg \'o \'z d \'z , T., Sankowski, P., et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07871, 2024

    2024 arXiv

  28. [36]

    S., Biderman, S., Elsahar, H., Muennighoff, N., Phang, J., Press, O., Raffel, C., Sanh, V., Shen, S., Sutawika, L., Tae, J., Yong, Z

    Le Scao, T., Wang, T., Hesslow, D., Bekman, S., Bari, M. S., Biderman, S., Elsahar, H., Muennighoff, N., Phang, J., Press, O., Raffel, C., Sanh, V., Shen, S., Sutawika, L., Tae, J., Yong, Z. X., Launay, J., and Beltagy, I. What language model to train if you have one million G...

    2022 doi

  29. [37]

    Gshard: Scaling giant models with conditional computation and automatic sharding

    Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding

  30. [38]

    M., Bartlett, P., and Lee, J

    Lin, L., Wu, J., Kakade, S. M., Bartlett, P., and Lee, J. D. Scaling laws in linear regression: Compute, parameters, and data. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

  31. [39]

    Logiqa: a challenge dataset for machine reading comprehension with logical reasoning

    Liu, J., Cui, L., Liu, H., Huang, D., Wang, Y., and Zhang, Y. Logiqa: a challenge dataset for machine reading comprehension with logical reasoning. pp.\ 3622--3628, 2021

    2021

  32. [40]

    J., Wang, S., Liang, C., Shen, Y., Cheng, H., Liu, X., Tanaka, M., Wu, X., Hu, W., et al

    Liu, L., Kim, Y. J., Wang, S., Liang, C., Shen, Y., Cheng, H., Liu, X., Tanaka, M., Wu, X., Hu, W., et al. Grin: Gradient-informed moe. arXiv preprint arXiv:2409.12136, 2024

    2024 arXiv

  33. [41]

    M., Huang, Z., Qiu, Z., Wang, Z., and Fu, J

    Lo, K. M., Huang, Z., Qiu, Z., Wang, Z., and Fu, J. A closer look into mixture-of-experts in large language models. arXiv preprint arXiv:2406.18219, 2024

    2024 arXiv

  34. [42]

    Fixing weight decay regularization in adam

    Loshchilov, I., Hutter, F., et al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5, 2017

    2017 arXiv

  35. [43]

    A., and Sully, J

    Maloney, A., Roberts, D. A., and Sully, J. A solvable model of neural scaling laws. arXiv preprint arXiv:2210.16859, 2022

    2022 arXiv

  36. [44]

    A scaling law for syn2real transfer: How much is your pre-training effective? pp.\ 477--492

    Mikami, H., Fukumizu, K., Murai, S., Suzuki, S., Kikuchi, Y., Suzuki, T., Maeda, S.-i., and Hayashi, K. A scaling law for syn2real transfer: How much is your pre-training effective? pp.\ 477--492. Springer, 2022

    2022

  37. [45]

    Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36: 0 50358--50376, 2023

    2023

  38. [46]

    Olmoe: Open mixture-of-experts language models

    Muennighoff, N., Soldaini, L., Groeneveld, D., Lo, K., Morrison, J., Min, S., Shi, W., Walsh, P., Tafjord, O., Lambert, N., et al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060, 2024

    2024 arXiv

  39. [47]

    The lambada dataset: Word prediction requiring a broad discourse context

    Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N.-Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern \'a ndez, R. The lambada dataset: Word prediction requiring a broad discourse context. Proceedings of the 54th Annual Meeting of the Association for Computatio...

    2016

  40. [48]

    4+ 3 phases of compute-optimal neural scaling laws

    Paquette, E., Paquette, C., Xiao, L., and Pennington, J. 4+ 3 phases of compute-optimal neural scaling laws. The Thirty-eighth Annual Conference on Neural Information Processing Systems

  41. [49]

    Resolving discrepancies in compute-optimal scaling of language models

    Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Carmon, Y. Resolving discrepancies in compute-optimal scaling of language models. Advances in Neural Information Processing Systems, 37: 0 100535--100570, 2024

    2024

  42. [50]

    S., Rosenfeld, A., Belinkov, Y., and Shavit, N

    Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. A constructive prediction of the generalization error across scales

  43. [51]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  44. [52]

    Glu variants improve transformer

    Shazeer, N. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  45. [53]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. 2017

    2017

  46. [54]

    Slimpajama-dc: Understanding data combinations for llm training

    Shen, Z., Tao, T., Ma, L., Neiswanger, W., Liu, Z., Wang, H., Tan, B., Hestness, J., Vassilieva, N., Soboleva, D., et al. Slimpajama-dc: Understanding data combinations for llm training. arXiv preprint arXiv:2309.10818, 2023

    2023 arXiv

  47. [55]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  48. [56]

    R., Hestness, J., and Dey, N

    Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama , June 2023. URL https://huggingface.co/datasets/cerebras/SlimPajama-627B

    2023

  49. [57]

    Roformer: Enhanced transformer with rotary position embedding

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  50. [58]

    Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent

    Sun, X., Chen, Y., Huang, Y., Xie, R., Zhu, J., Zhang, K., Li, S., Yang, Z., Han, J., Shu, X., et al. Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent. arXiv preprint arXiv:2411.02265, 2024

    2024 arXiv

  51. [59]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  52. [60]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  53. [61]

    Skywork-moe: A deep dive into training techniques for mixture-of-experts language models

    Wei, T., Zhu, B., Zhao, L., Cheng, C., Li, B., L \"u , W., Cheng, P., Zhang, J., Zhang, X., Zeng, L., et al. Skywork-moe: A deep dive into training techniques for mixture-of-experts language models. arXiv preprint arXiv:2406.06563, 2024

    2024 arXiv

  54. [62]

    F., and Gardner, M

    Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. W-NUT 2017, pp.\ 94, 2017

    2017

  55. [63]

    Qwen2 technical report

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  56. [64]

    Glm-130b: An open bilingual pre-trained model

    Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022

    2022 arXiv

  57. [65]

    Scaling vision transformers

    Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. pp.\ 12104--12113, 2022

    2022

  58. [66]

    When scaling meets llm finetuning: The effect of data, model and finetuning method

    Zhang, B., Liu, Z., Cherry, C., and Firat, O. When scaling meets llm finetuning: The effect of data, model and finetuning method

  59. [67]

    Tinyllama: An open-source small language model

    Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024

    2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.