Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Chain-of-Experts claims that processing tokens through a sequence of expert selections within a single MoE layer—rather than one parallel selection—improves performance at fixed compute, reducing math validation loss from 1.20 to 1.12 and…

desk verdict A promising architecture idea whose central communication claim is not yet isolated from depth, residual, and routing confounds. read the letter →

arxiv 2506.18945 v1 pith:7OKQMHXY submitted 2025-06-23 cs.LG cs.CL

Zihan Wang , Rui Pan , Jiarui Yao , Robert Csordas , Linjie Li , Lu Yin , Jiajun Wu , Tong Zhang
show 2 more authors
Manling Li Shiwei Liu
This is my paper
classification cs.LGcs.CL
keywords Chain-of-ExpertsMixture-of-Expertssequentialexpertcommunicationiterativeroutingsparseeffectivedepthspecializationscalingaxis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Chain-of-Experts (CoE) claims that the standard mixture-of-experts (MoE) layer, which activates a fixed set of experts in parallel for each token, leaves performance on the table. By processing the same token through a sequence of expert selections within one layer—each round with its own router and a residual connection—CoE achieves lower validation loss at the same per-token expert compute: on math reasoning, loss drops from 1.20 to 1.12. The paper further argues that increasing the number of sequential rounds (depth through expert iteration) is a more memory-efficient scaling axis than widening the expert set or deepening the network. If true, the result changes where to spend compute in sparse language models: expert communication, not just expert count, is a first-class resource.

What carries the argument

The load-bearing mechanism is the iterative CoE layer update $x^{(t)} = \sum_{i=1}^N g_{t,i}\,E_i(x^{(t-1)}) + x^{(t-1)}$, with an independent router $g_{t,i}$ at each communication step $t$ that selects $K/C$ experts. This identity converts one parallel mixture into a chain of conditional transformations, letting later experts see and refine earlier experts' outputs. Its effectiveness rests on two design choices the ablations isolate: per-iteration gating (shared gating across steps collapses validation loss to about 1.5) and inner residuals (an outer-only residual raises loss from 1.12 to 1.21). The combinatorial corollary—two sequential top-$k$ choices give $\binom{N}{k}^2$ combinations instead of $\binom{N}{2k}$—quantifies the added routing diversity.

What would settle it

Train CoE (K=4,C=2) and MoE (K=8,C=1) with identical total router forward passes and an equal number of residual connections (for example, give MoE a parallel inner residual or run CoE's routers all on the original input). If the validation-loss gap vanishes or reverses, sequential communication is not the operative mechanism. Alternatively, count the distinct expert sequences actually taken by tokens at convergence: if the number is close to the MoE's single-step combinations, the combinatorial-diversity claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that a sparse MoE layer can be reorganized from a one-shot parallel computation into an iterative relay race without increasing the number of expert invocations, and this reorganization alone improves generalization. Formally, CoE replaces the MoE output weighted sum of K experts with C sequential rounds, where each round selects K/C experts using its own router and adds the previous round's hidden state as an inner residual. At matched compute, CoE (K=4,C=2) reduces math validation loss from 1.20 to 1.12 relative to MoE (K=8,C=1), and CoE with 2 iterations matches the performance of MoE with 3 times the expert selections while using 17.6–42% less memory. The paper attributes the gains to two mechanisms: combinatorial flexibility—two separate top-k choices yield many more expert pairings than one top-2k choice—and implicit depth, since each round refines the representation conditioned on intermediate states. Analyses of co-activation patterns show the per-round routers specialize on different experts, supporting the claim that sequential communication, not merely more parameters, drives the improvement.

Load-bearing premise

The central premise is that the experiments isolate sequential expert communication under fixed compute, but the CoE and MoE configurations differ in several other ways at once—the number of routing decisions, the presence of an additional residual path, total parameters, layer count, and attention depth—so if any of these confounds drives the loss reduction, the core claim does not follow.

Editorial extensions

If this is right

  • Under the paper's fixed-compute comparisons, standard MoE layers are suboptimal; sequential expert chains should be considered a default architectural option for sparse models.
  • Scaling an MoE by communication steps is cheaper than scaling by width or depth: 2× iterations match 3× expert selections with 17.6–42% lower memory.
  • Iteration-specific routing yields measurable expert specialization, as seen in asymmetric co-activation matrices between communication steps.
  • The combinatorial diversity argument implies that even at the same expert budget, CoE explores a much larger space of expert compositions, which may explain faster convergence.
  • The design requires training from scratch and adds wall-clock overhead despite FLOP parity, so its efficiency benefit is asymptotic rather than immediate on current hardware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The loss reduction could be partly an optimization artifact of having more, fresher routing decisions and an extra residual path; a head-to-head run that equalizes the number of gating forward passes and residual connections would test whether communication itself, rather than added capacity, is the cause.
  • CoE's mechanism suggests a natural combination with cross-layer expert sharing or test-time chain-of-thought-style refinement: the same iterative residual architecture could make reasoning depth a tunable inference-time knob.
  • Because sequential expert execution reduces matrix-multiplication parallelism, CoE's gains are likely to grow on hardware or kernels that lower sequential-step overhead (e.g., fused schedules), making the architecture a co-design target.
  • The 823× combination-count figure may overstate practical diversity if the routers collapse to a few favored paths; measuring the empirical number of distinct expert chains used at convergence would bound the true combinatorial benefit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Chain-of-Experts (CoE), a modification of Mixture-of-Experts layers in which a token is processed sequentially over C communication steps, with a dedicated router at each step and an inner residual connection. The central empirical claim is that, at equal per-token expert invocation count, CoE with K=4, C=2 reduces math validation loss from 1.20 to 1.12 compared to a standard MoE with K=8, C=1 (Section 5.1). The paper also presents scaling experiments suggesting that increasing C matches or beats increasing depth, width, or expert count at lower memory (Section 5.2); ablations of iteration-specific gating and inner residuals (Section 5.3); co-activation analyses (Section 5.4); and a combinatorial flexibility argument (Section 5.5). The authors acknowledge practical limitations in Section 8, including sequential processing overhead, single-device testing, and incompatibility with pretrained MoE checkpoints.

Significance. The CoE idea is a natural and potentially valuable architectural variant: intra-layer iterative expert routing with per-step routers and residuals could provide a new compute-efficiency axis for sparse models. The paper is honest about several limitations and the experiments are small enough to be reproducible within about one GPU hour per run. If the empirical gains are robust and genuinely attributable to sequential expert communication, this would be a useful contribution to MoE architecture design. However, as presented, the central mechanistic claim is underdetermined: the headline comparison varies the number of routing decisions, the depth of the computation graph, and the residual structure simultaneously, and no non-communicative same-depth control is reported. The combinatorial argument, which is featured in the abstract and introduction, contains a numerical error. The contribution is significant conditional on the additional controls and corrections described below.

major comments (5)
  1. [§5.1 and Eq. (5)-(7)] The headline comparison CoE (K=4, C=2) versus MoE (K=8, C=1) varies more than the communication structure: CoE applies two sequential expert transformations with an inner residual at each step and two independent routers, whereas MoE applies one parallel expert combination with a single router. The paper's own ablations in Figure 7 show that removing either the inner residual or the iteration-specific router degrades performance substantially, so those components are doing real work. A control that keeps the same number of expert invocations, the same two K=4 routing computations, and the same residual structure but makes the second step independent of the first step's hidden state (for example, by routing the second step from the original input) is needed to attribute the gain to sequential communication rather than to extra depth, extra routing decisions, or the residual. Without such a control, the central mechanistic claim that communication between experts drives the improvement is underdetermined.
  2. [§5.5 and Abstract] The combinatorial claim that CoE with n=64 and k=4 leads to "823× more expert pairings" is numerically incorrect: C(64,4)^2 / C(64,8) is approximately 91, not 823. Moreover, even with the correct ratio, the counting argument is an upper bound that ignores the fact that the second router is conditioned on the first step's output, so the model cannot realize all C(n,k)^2 ordered combinations independently. This point should be corrected in the abstract, introduction, and Section 5.5, and the argument should be reframed as a hypothesis about flexibility rather than a derived advantage.
  3. [§5.2, Figures 4-6] The scaling comparisons vary more than the communication step count. In Figure 4, a 4-layer CoE (C=2) is compared to 8- and 12-layer MoEs, so total parameters, layer count, and attention depth change simultaneously with C. Similarly, Figure 6 compares CoE with N=48 experts to MoE with N=64, changing both architecture and expert count. These experiments support the practical efficiency of CoE but do not isolate C as a scaling axis. The paper should either add matched-parameter and matched-depth baselines or explicitly reframe these results as engineering trade-offs rather than controlled scaling-law evidence.
  4. [§5.1 and §8] The phrase "fixed compute" is used in the abstract and Section 5.1 to mean equal per-token expert invocation count, but Section 8 acknowledges that CoE "introduces moderate time overhead in practice despite theoretical FLOP parity" due to sequential processing. The paper should distinguish expert-invocation FLOPs from wall-clock compute and report actual training or inference throughput for the compared configurations; otherwise, the efficiency claims in the abstract and Section 5.2 are stronger than what the experiments demonstrate.
  5. [Figures 3-6 and Table 1] All reported loss curves and benchmark numbers appear to come from single runs; no error bars, multiple seeds, or variance estimates are given. Given the small model scale and short training runs, it is unclear whether the headline 1.20 to 1.12 loss gap and the small benchmark differences in Table 1 are statistically reliable. The paper should report at least three seeds with mean and standard deviation for the main comparisons, or clearly state that the results are single-run observations.
minor comments (6)
  1. [§3 and §4.3] The meaning of K is ambiguous: in Eq. (7) the text says K/C experts are selected per iteration, but in Section 5.2 the text refers to "CoE retains K=8" alongside C=2, and in Section 4.3 it says the router selects 8 routed experts under a C=2 setup. Please define whether K denotes per-iteration selection, total selection, or both, and use consistent notation such as K_total = C * K_per_iter.
  2. [Eq. (5), (8), (9)] The model in Section 4.3 includes one shared expert, and Eq. (8) and (9) include a shared-expert term, but Eq. (5) and (7) omit it. Please add the shared-expert term to the main formulation or state explicitly when shared experts are not used.
  3. [References] References [17] and [32] are the same paper (Wang et al., "Let the expert stick to his last"); one of them should be removed or replaced with a distinct citation.
  4. [§5.4, Figure 8] The co-activation analysis is qualitative. The claim of step-wise specialization would be strengthened by a quantitative diversity or transition-entropy metric, and by a comparison to a random or independent-routing baseline.
  5. [Abstract] The abstract says "iterative" in the phrase "processes tokens iterative across a chain"; this should be "iteratively."
  6. [§5.5] The "effective depth" discussion cites chain-of-thought and recurrent-depth literature but does not define a formal measure of effective depth for CoE. Consider stating a precise sense in which C communication steps increase representational depth relative to a one-step MoE.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: CoE's claims are empirical comparisons with standard MoE baselines; no fitted parameter, definition, or self-citation chain forces the reported results.

full rationale

The paper's central claims are empirical: CoE (K=4,C=2) versus MoE (K=8,C=1) is an experimental comparison under matched expert-invocation count. CoE's update (Eq. 5) and routing (Eq. 7) are defined independently of the reported losses; the loss values (1.20 vs 1.12) come from training runs, not from the equations. The combinatorial flexibility argument in Sec. 5.5 is a count of possible ordered expert subsets versus unordered ones; it is an independent combinatorial fact, though the 823x ratio is numerically incorrect, which is a correctness concern rather than circularity. The ablations in Sec. 5.3 test whether iteration-specific gating and inner residuals matter; these are genuine component-level tests, not restatements of the definition. The self-citation [17] is used only as motivation ('experts often learn complementary roles...') and is not load-bearing for any quantitative claim. The paper is self-contained against standard MoE baselines; no equation assumes its own conclusion.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities are proposed. The only quantities chosen by hand are architectural hyperparameters C, K/C, and the inner residual weight. The central quantitative claims are empirical, so the ledger is light; load-bearing assumptions are about what 'fixed compute' means and about dataset representativeness.

free parameters (3)
  • communication steps C = 2 (hand-selected)
    Main results use C=2; supplement shows C=3,4 give diminishing or unstable gains. The headline comparison depends on this choice.
  • per-iteration experts K/C = 4
    Set so that total expert invocations per token match baseline MoE (K=8). This controls expert FLOPs but also changes routing granularity and adds a second routing decision.
  • inner residual coefficient I_r = 1
    Residual is added with fixed weight 1 at every iteration; no tuning is reported.
assumptions (4)
  • standard math Top-k softmax routing is differentiable and trainable end-to-end with standard backpropagation.
    Used throughout Equations 5 and 7; no formal treatment of gradient flow through iterative routing is given.
  • domain assumption Validation loss on held-out slices of MetaMathQA and SlimPajama is a meaningful proxy for language modeling and reasoning quality.
    Section 4.1; the paper does not report reasoning benchmarks such as GSM8K or MATH, and downstream accuracies are near chance.
  • ad hoc to paper Matching per-token expert invocation count (K per token) makes CoE and MoE comparable in compute, despite additional routers, residuals, and sequential execution.
    Section 5.1 and Abstract; FLOP parity ignores router overhead, reduced parallelism, and attention-depth differences in the scaling comparisons of Figures 4-6.
  • domain assumption Backpropagation through C unrolled iterations is stable with inner residual connections.
    Section 5.3 and Eq. 5; ablation shows outer residual causes worse convergence, but no stability analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models." pith.science (2026). https://pith.science/paper/7OKQMHXY

@misc{pith2026250618945,
  author       = {Pith},
  title        = {Pith review of: Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7OKQMHXY}},
  note         = {Machine review of arXiv:2506.18945}
}
read the original abstract

We propose Chain-of-Experts (CoE), a new Mixture-of-Experts (MoE) architecture that introduces sequential expert communication within each layer. Unlike traditional MoE models, where experts operate independently in parallel, CoE processes tokens iteratively across a chain of experts inside a layer. To support dynamic expert selection across iterations, CoE employs a dedicated router at each iteration step within a layer. This design allows tokens to re-evaluate and select different experts during each iteration, rather than being statically assigned. As a result, CoE introduces a flexible routing mechanism that increases the diversity of expert combinations and enriches the model's representational capacity. CoE demonstrates improved performance under fixed compute: on math reasoning tasks, it reduces validation loss from 1.20 to 1.12 compared to a standard MoE. Beyond performance, CoE offers a new scaling axis: depth through expert iteration, which complements conventional width/depth scaling. For example, using 2x iterations matches the performance of 3x expert selections (in width), while reducing memory usage by 17.6-42% relative to other scaling strategies. Our analysis reveals that CoE's benefits stem from its iterative residual structure and enhanced expert specialization empowered by iterative routing, which together unlock more expressive representations. Code is available at https://github.com/ZihanWang314/coe.

Figures

Figures reproduced from arXiv: 2506.18945 by the authors.

Figure 1
Figure 1. Comparison between Mixture-of-Experts (MoE) and Chain-of-Experts (CoE). Under the same model depth and parameters, CoE enables iterative expert communication to offer more flexible expert choice compared to MoE. experts. As a result, existing MoE models could underutilize their available capacity, particularly for complex tasks that benefit from multi-expert coordination. We challenge this independence assumption by… view at source ↗
Figure 2
Figure 2. Illustration of Chain-of-Experts. MoE Top-K experts operate in parallel without inter￾action, CoE enables the same number of experts to process sequentially via intermediate represen￾tations, allowing deeper network processing with the same per-token expert processing. Residual connections are enabled and iteration-independent routers are used for more effective training. mechanism ensures sparsity: only K experts a… view at source ↗
Figure 3
Figure 3. CoE reduces validation loss more effectively than MoE under equal expert compute, specifically in sparse settings. CoE (K=4, C=2) outperforms MoE (K=8, C=1) with the same per-token expert processing (left). The benefit is amplified in sparse routing, where communication fosters specialization, but diminishes in dense settings where all experts are active (right). 4.4 System and Infrastructure All experiments are imp… view at source ↗
Figures from the paper (19 more)
Figure 4
Figure 4. Figure 4: Depth scaling comparison. CoE (C=2) with 4 layers matches the performance of deeper MoE models (L=8 or 12) with significantly lower memory and parameter usage [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Width scaling comparison. CoE (C=2) outperforms MoE variants with increased expert selection (K=16 or 24), while using similar resources. 5.1 Communicative Processing can be Better than Parallel Processing in MoEs We begin by discussing whether communicative processing…
Figure 6
Figure 6. Figure 6: Matched performance with fewer experts. CoE (N=48) achieves similar performance to MoE (N=64) while reducing memory usage by 17.6%. number of transformer layers from 4 to 8 and 12. We compare these models to a 4-layer CoE model with C=2 expert communication steps. As s…
Figure 7
Figure 7. Figure 7: Ablation study. Removing either iteration-specific gating or inner residuals significantly reduces performance. Both components are critical to the effectiveness of sequential expert routing in CoE. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Expert co-activation between communication steps. For each selected layer, we plot a matrix counting how often each expert pair (e0, e1) is activated across the two communication steps in CoE. The non-uniform, asymmetric patterns indicate that routing decisions differ …
Figure 9
Figure 9. Figure 9: Validation loss on MetaMathQA across 10,000 steps. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Validation loss on general-domain SlimPajama across 10,000 steps. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Layer-wise routing pattern on MetamathQA under the 4-experts-per-iteration, 2-iteration [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Layer-wise routing pattern on MetamathQA under the 4-experts-per-iteration, 2-iteration [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Layer-wise routing pattern on SlimPajama under the 4-experts-per-iteration, 2-iteration [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Layer-wise routing pattern on SlimPajama under the 4-experts-per-iteration, 2-iteration [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Layer-wise routing pattern on MetamathQA under the 8-experts-per-iteration, 2-iteration [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Layer-wise routing pattern on MetamathQA under the 8-experts-per-iteration, 2-iteration [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: Layer-wise routing pattern on SlimPajama under the 8-experts-per-iteration, 2-iteration [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: Layer-wise routing pattern on SlimPajama under the 8-experts-per-iteration, 2-iteration [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Layer-wise routing pattern on MetamathQA under the 8-experts-per-iteration, 1-iteration [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]
Figure 20
Figure 20. Figure 20: Layer-wise routing pattern on MetamathQA under the 8-experts-per-iteration, 1-iteration [PITH_FULL_IMAGE:figures/full_fig_p021_20.png]
Figure 21
Figure 21. Figure 21: Layer-wise routing pattern on SlimPajama under the 8-experts-per-iteration, 1-iteration [PITH_FULL_IMAGE:figures/full_fig_p021_21.png]
Figure 22
Figure 22. Figure 22: Layer-wise routing pattern on SlimPajama under the 8-experts-per-iteration, 1-iteration [PITH_FULL_IMAGE:figures/full_fig_p021_22.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mixture of Raytraced Experts

    cs.LG 2025-07 conditional novelty 7.0 of 10

    A stacked Mixture-of-Experts model that dynamically selects ordered sequences of experts via a ray-like stochastic routing process, improving accuracy as more experts are activated.

Reference graph

Works this paper leans on

51 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeffrey Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InICLR, 2017

    work page 2017

  2. [2]

    Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020

    arXiv 2006

  3. [3]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.JMLR, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.JMLR, 23(120):1–39, 2022

    work page 2022

  4. [4]

    Designing effective sparse expert models

    Barret Zoph. Designing effective sparse expert models. In2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pages 1044–1044, 2022. doi: 10.1109/IPDPSW55747.2022.00171

    arXiv 2022

  5. [5]

    Mixture-of-experts with expert choice routing,

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon. Mixture-of-experts with expert choice routing,

    work page

  6. [7]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  7. [8]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation,

    Meta. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation,

  8. [9]

    Qwen3: Think deeper, act faster, 2025

    Qwen Team. Qwen3: Think deeper, act faster, 2025. URLhttps://qwenlm.github.io/ blog/qwen3/

    work page 2025

Show all 51 references
  1. [10]

    Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  2. [11]

    Olmoe: Open mixture-of- experts language models.arXiv preprint arXiv:2409.02060, 2024

    Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of- experts language models.arXiv preprint arXiv:2409.02060, 2024

    2024 arXiv

  3. [12]

    Scaling vision with sparse mixture of experts, 2021

    Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andr´e Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts, 2021. URLhttps://arxiv.org/abs/2106.05974

    2021 arXiv

  4. [13]

    Uni-moe: Scaling unified multimodal llms with mixture of experts, 2024

    Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. Uni-moe: Scaling unified multimodal llms with mixture of experts, 2024. URLhttps://arxiv.org/abs/2405.11273

    2024 arXiv

  5. [14]

    Sparse universal transformer.arXiv preprint arXiv:2310.07096, 2023

    Shawn Tan, Yikang Shen, Zhenfang Chen, Aaron Courville, and Chuang Gan. Sparse universal transformer.arXiv preprint arXiv:2310.07096, 2023

    2023 arXiv

  6. [15]

    Moeut: Mixture-of-experts universal transformers.arXiv preprint arXiv:2405.16039, 2024

    R ´obert Csord´as, Kazuki Irie, J¨urgen Schmidhuber, Christopher Potts, and Christopher D Man- ning. Moeut: Mixture-of-experts universal transformers.arXiv preprint arXiv:2405.16039, 2024

    2024 arXiv

  7. [16]

    Lay- erwise recurrent router for mixture-of-experts.arXiv preprint arXiv:2408.06793, 2024

    Zihan Qiu, Zeyu Huang, Shuang Cheng, Yizhi Zhou, Zili Wang, Ivan Titov, and Jie Fu. Lay- erwise recurrent router for mixture-of-experts.arXiv preprint arXiv:2408.06793, 2024

    2024 arXiv

  8. [18]

    Deepseekmoe: Towards ultimate expert special- ization in mixture-of-experts language models.arXiv preprint arXiv:2401.06066, 2024

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert special- ization in mixture-of-experts language models.arXiv preprint arXiv:2401.06066, 2024

    2024 arXiv

  9. [19]

    Uni- versal transformers

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Uni- versal transformers. InInternational Conference on Learning Representations, 2019

    2019

  10. [20]

    SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.https://www.cerebras.net/blog/ slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama,

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.https://www.cerebras.net/blog/ slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama,

  11. [21]

    Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2024. URLhttps://arxiv.org/abs/2309.12284

    2024 arXiv

  12. [22]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv:1803.05457v1, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv:1803.05457v1, 2018

    2018 arXiv

  13. [23]

    Hellaswag: Can a machine really finish your sentence?, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URLhttps://arxiv.org/abs/1905.07830. 11

    2019 arXiv

  14. [24]

    Piqa: Reasoning about physical commonsense in natural language, 2019

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. URLhttps://arxiv.org/abs/ 1911.11641

    2019 arXiv

  15. [25]

    Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei...

    2024 arXiv

  16. [26]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  17. [27]

    Towards revealing the mystery behind chain of thought: A theoretical perspective, 2023

    Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: A theoretical perspective, 2023. URLhttps: //arxiv.org/abs/2305.15408

    2023 arXiv

  18. [28]

    A little depth goes a long way: The expressive power of log-depth transformers, 2025

    William Merrill and Ashish Sabharwal. A little depth goes a long way: The expressive power of log-depth transformers, 2025. URLhttps://arxiv.org/abs/2503.03961

    2025

  19. [29]

    Bar- toldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein

    Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bar- toldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time com- pute with latent reasoning: A recurrent depth approach.arXiv preprint arXiv:2502.05171, 2025

    2025 arXiv

  20. [30]

    Adaptive mix- tures of local experts.Neural computation, 3(1):79–87, 1991

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mix- tures of local experts.Neural computation, 3(1):79–87, 1991

    1991

  21. [31]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  22. [32]

    Zihan Wang, Deli Chen, Damai Dai, Runxin Xu, Zhuoshu Li, and Y . Wu. Let the expert stick to his last: Expert-specialized fine-tuning for sparse architectural large language models, 2024. URLhttps://arxiv.org/abs/2407.01906

    2024 arXiv

  23. [33]

    Skywork-moe: A deep dive into training techniques for mixture-of-experts language models.arXiv preprint arXiv:2406.06563, 2024

    Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng, Biye Li, Weiwei L ¨u, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Liang Zeng, et al. Skywork-moe: A deep dive into training techniques for mixture-of-experts language models.arXiv preprint arXiv:2406.06563, 2024

    2024 arXiv

  24. [34]

    End-to-end algorithm synthesis with recurrent networks: Ex- trapolation without overthinking

    Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Gold- blum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Ex- trapolation without overthinking. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and 12 Kyunghyun Cho, ...

    2022

  25. [35]

    Rethinking deep thinking: Stable learn- ing of algorithms using lipschitz constraints, 2024

    Jay Bear, Adam Pr ¨ugel-Bennett, and Jonathon Hare. Rethinking deep thinking: Stable learn- ing of algorithms using lipschitz constraints, 2024. URLhttps://arxiv.org/abs/2410. 23451

    2024

  26. [36]

    Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, and Tom Goldstein

    Sean Michael McLeish, Arpit Bansal, Alex Stein, Neel Jain, John Kirchenbauer, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, and Tom Goldstein. Transformers can do arithmetic with the right embeddings. InThe Thirty- eighth Annual Conf...

    2024

  27. [37]

    Adaptive computation time for recurrent neural networks.arXiv preprint arXiv:1603.08983, 2016

    Alex Graves. Adaptive computation time for recurrent neural networks.arXiv preprint arXiv:1603.08983, 2016

    2016 arXiv

  28. [38]

    Albert: A lite bert for self-supervised learning of language representations

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. InInter- national Conference on Learning Representations (ICLR), 2020

    2020

  29. [39]

    Zico Kolter, and Vladlen Koltun

    Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. InAdvances in Neural Information Processing Systems, volume 32, 2019

    2019

  30. [40]

    Chain-of-thought prompting elicits reasoning in large language models.arXiv preprint arXiv:2201.11903, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models.arXiv preprint arXiv:2201.11903, 2022

    2022 arXiv

  31. [41]

    When reasoning meets compres- sion: Benchmarking compressed large reasoning models on complex reasoning tasks.arXiv preprint arXiv:2504.02010, 2025

    Nan Zhang, Yusen Zhang, Prasenjit Mitra, and Rui Zhang. When reasoning meets compres- sion: Benchmarking compressed large reasoning models on complex reasoning tasks.arXiv preprint arXiv:2504.02010, 2025

    2025

  32. [42]

    Jeffrey L. Elman. Finding structure in time.Cognitive Science, 14(2):179–211, 1990. ISSN 0364-0213. doi: https://doi.org/10.1016/0364-0213(90)90002-E. URLhttps://www. sciencedirect.com/science/article/pii/036402139090002E

    1990

  33. [43]

    Learning to control fast-weight memories: An alternative to dynamic recurrent networks.Neural Computation, 4(1):131–139, 1992

    J ¨urgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks.Neural Computation, 4(1):131–139, 1992. doi: 10.1162/neco.1992.4.1. 131

    1992 doi

  34. [44]

    Long short-term memory.Neural Comput., 9 (8):1735–1780, November 1997

    Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory.Neural Comput., 9 (8):1735–1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL https://doi.org/10.1162/neco.1997.9.8.1735

    1997 doi

  35. [45]

    Lessons on parameter sharing across layers in transformers

    Sho Takase and Shun Kiyono. Lessons on parameter sharing across layers in transformers. In Nafise Sadat Moosavi, Iryna Gurevych, Yufang Hou, Gyuwan Kim, Young Jin Kim, Tal Schuster, and Ameeta Agrawal, editors,Proceedings of the Fourth Workshop on Simple and Efficient Natural ...

    2023 doi

  36. [46]

    Depth-adaptive transformer

    Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In International Conference on Learning Representations (ICLR), 2020

    2020

  37. [47]

    Reducing transformer depth on demand with structured dropout

    Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. InInternational Conference on Learning Representations (ICLR), 2020

    2020

  38. [48]

    So- lar 10.7b: Scaling large language models with simple yet effective depth up-scaling, 2024

    Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeon- woo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, and Sunghun Kim. So- lar 10.7b: Scaling large lan...

    arXiv

  39. [49]

    The devil is in the detail: Simple tricks improve systematic generalization of transformers

    R ´obert Csord ´as, Kazuki Irie, and Juergen Schmidhuber. The devil is in the detail: Simple tricks improve systematic generalization of transformers. In Marie-Francine Moens, Xuan- jing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Proceedings of the 2021 Confer- ence o...

    2021 doi

  40. [50]

    Self-delimiting neural networks, 2012

    Juergen Schmidhuber. Self-delimiting neural networks, 2012. URLhttps://arxiv.org/ abs/1210.0118. 14 A Extended Related Works Mixture-of-Experts (MoE).MoE architectures extend neural capacity through conditional com- putation [30], activating only a small subset of experts per ...

    2012 arXiv

  41. [2022]

    URLhttps://arxiv.org/abs/2202.09368

    arXiv

  42. [2023]

    URLhttps://huggingface.co/datasets/cerebras/SlimPajama-627B

  43. [2025]

    URLhttps://ai.meta.com/blog/llama-4-multimodal-intelligence/

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.